Multimodal Methodology Research Articles

The Dark Horse: Introducing a Comparative Hematology Course into the Post-Clerkship Phase of Medical School

Introduction: Developing a deep understanding of pathophysiology is integral to training competent physicians. Although the basic sciences are taught primarily in preclinical curriculum, revisiting these concepts during the fourth year of medical school promotes integration into clinical medicine (Spencer et. al, 2008). Immersive comparative physiology courses increase trainee appreciation of physiology and its incorporation into patient care (Herzig et al, 2012). We introduced comparative physiology curriculum during a fourth-year medical student course at University of Texas Southwestern (UTSW) Medical School. Our course aimed to deepen students' understanding of human pathophysiology and its clinical applications by highlighting evolutionarily conserved and adaptive changes among diverse species. Methods: We developed a one-week comparative physiology curriculum to present to fourth-year medical students in April 2024. The curriculum utilized the four stages of Kolb's Experiential Learning Theory through multimodal methodologies, including introductory lectures, case-based learning, laboratory experiments on human and animal blood with integrated case studies and group projects, and evolutionary biology group presentations at the Fort Worth Zoo. The UTSW Institutional Review Board approved the dissemination of pre- and post-course surveys. The surveys incorporated a 7-point Likert scale (1= strongly disagree, 2= disagree, 3= somewhat disagree, 4= neither agree nor disagree, 5= somewhat agree, 6= agree, 7= strongly agree). Anonymous six-digit identifiers tracked changes in individual perceptions. Scores were calculated by summing responses to series of questions evaluating perceptions of construct and knowledge. Paired samples t-tests through Microsoft Excel assessed the significance of differences between pre- and post-course mean perception of construct and confidence in knowledge scores. Results: Fifteen fourth-year medical students enrolled, and 93% completed both surveys. Self-assessed confidence in knowledge significantly increased (p =1.02x10-5) in multiple concepts, including interpretation of the peripheral blood smear and of the complete blood count. In the post-course survey, 79% of respondents agreed or strongly agreed that comparative physiology models should be incorporated into medical education, 79% agreed or strongly agreed that the course increased their interest in hematology, and 100% strongly agreed to recommend the course. Conclusion: Incorporating immersive comparative physiology curriculum into the post-clerkship phase of medical school can enhance understanding of core hematologic physiology principles and promote interest in hematology.

Mapping the Path to Low-Carbon Behaviour: A Systematic Review of Trends, Gaps, and Future Directions

It is essential to conduct research on low-carbon behaviour in order to address global climate change and promote sustainable development. This study conducts a thorough examination of a bibliometric analysis of the existing literature in this field. We analysed 129 papers from the Web of Science Core Collection database to conduct a study on the concept of “low-carbon behaviour”. Review articles, proceedings papers, and editorial materials were all excluded. This investigation examines a diverse array of research environment factors, including the most highly ranked publications, publication trends, significant publishers, and patterns in citations and publications over time. The primary institutional contributors in the discipline and the most influential works worldwide are also mentioned. The application of advanced visualisation techniques, such as wordcloud representations, the spatial distribution of research output, and co-occurrence and overlay networks, is employed to generate maps of keyword relationships and collaboration networks using Bibliometrix Stable Version, VOSviewer 1.6.20, and Scimago Graphica Beta 1.0.45. This multi-modal methodology enables a comprehensive investigation of significant research issues and emerging directions in low-carbon behavioural studies. This study contributes to the ongoing endeavours to promote sustainable development and mitigate climate change by conducting a comprehensive evaluation of the current state of research and establishing a robust framework for future investigations into low-carbon behavioural patterns and behaviours.

Not burning, but rebuilding: The ABCS of anthropology using multimodal methodologies and DEAF values

In 2019, Ryan Cecil Jobson made the case for ‘letting anthropology burn’, arguing that: “the field of anthropology cannot presume a coherent human subject as its point of departure but must adopt a radical humanism as its political horizon” (2019:328). In this article, Robinson upholds Jobson’s mandate to adopt a radical humanism, though not by burning, but methodologically rebuilding anthropology using multimodal ethnographic praxis, drawing on learnings from time spent working and learning from DEAF spaces. She charts her intellectual and affective journey from working with deaf artists and educators, through studies in anthropology, and towards transformative encounters with multimodal social semiotics. Through learning to listen and evaluate in DEAF ways, she found that non-deaf, majority-centred ways of conducting research naturally receded, thereby re-scaffolding ethnographic methods, proffering a ready-made approach to inclusive anthropological knowledge-making and a potential means for democratising anthropology from the ground up. Robinson argues that approaching ethnographic praxis in a way that both de-prioritises English-text and prioritises participant-led approaches by default reshapes what constitutes valuable anthropological knowledge, facilitating broader participant inclusion and legibility, while also rendering more apparent entrenched power hierarchies and epistemic imbalances.

Towards a data platform for multimodal 4D mechanics of material microstructures

This paper presents advances in the data management strategy applied to 4D multimodal mechanics of material sample microstructures. Guidelines to build a data platform allowing for complex workflows, involving several high-throughput experimental and numerical techniques, and complying with FAIR data management principles, are discussed. Next, their implementation within the open-source Python package Pymicro is presented, offering a high-level interface to build complex datasets through multimodal methodologies. Its capability to enable and automate complex workflows by building a digital twin of a commercially pure titanium sample under tension are then demonstrated. The digital twin contains microstructural and mechanical data for thousands of grains gathered on the same sample through synchrotron DCT and in-situ SEM experiments, as well as full-field numerical simulation. Finally, a local and statistical comparison between simulation and measurements of plastic slip and crystal rotation in hundreds of grains is shown, as an example of the contribution of this platform to multimodal data convergence and its importance for the development of a new generation of material models.

Novel multi-modal methodology to investigate placebo response in major depressive disorder

The neurobiological mechanisms underlying the placebo phenomenon in patients with major depressive disorder (MDD) remain largely unknown. The progressive rise in rates of placebo responses within clinical trials over the past two decades may impede the detection of a true signal and thus present a major obstacle in new treatment development. Understanding the mechanisms would have several important implications, including (1) identifying biomarkers of placebo responders (thereby identifying those individuals who could benefit therapeutically from such interventions), (2) opening new avenues for manipulating such mechanisms to maximize symptom reduction, and (3) refining treatments with approaches that decrease (in clinical trials) or increase (in clinical practice) the placebo response. Here we investigated the research question: is the dopaminergic system one of the neurobiological underpinnings of the placebo response within MDD? Inspired by preclinical and clinical findings that have implicated dopamine in the occurrence, prediction, and expectation of reward, we hypothesized that dopaminergic activity in the mesolimbic system is a critical mediator of placebo response in MDD. To test this hypothesis, we designed a double-blind, placebo-controlled, sequential parallel comparison design clinical trial aimed at maximizing placebo antidepressant response. We integrated behavioral, imaging, and hemodynamic probes of mesocorticolimbic dopaminergic pathways within the context of manipulations of psychological constructs previously linked to placebo responses (e.g., expectation of improvement). The aim of this manuscript is to present the rationale of the study design and to demonstrate how a cross-modal methodology may be utilized to investigate the role of reward circuitry in placebo response in MDD.

Understanding meaning-making through technology use

Previous research has highlighted that technology implementation in schools may lead to increased complexity, as digital hardware and software offer a variety of possibilities for sign-making activities. Moreover, recent studies argue that since classroom practices are facilitated increasingly by screen-based activities, digital technology opens a multitude of ways to represent meaning, as an abundance of sign systems becomes available for communication through various digital visual user interfaces (DVUIs) (Jewitt, 2017). In addition, research indicates that technology implementation has a strong impact on school practice (Säljö, 2013) and that knowledge on how to take advantage of technology in learning settings from a more comprehensive perspective is needed (Bezemer & Kress, 2016). To gain a more comprehensive picture of technology use in educational environments, the main goal of the thesis is to explore the use of hardware and software by teachers and students in sign-making activities from a multimodal layer perspective. The main aim of this paper is, in particular, to discuss how multimodal methodology can be used to explain detailed aspects of technology use in networked learning (NL) settings. Concerning the various means used in school and their affordances in semiotic mediation (Norman, 2007; Wartofsky, 1979), all are considered in relation to the users and results of use. From a technology perspective, the multimodal layers, therefore, include things-to-things, things[1]to-human/human-to-things and human-to-human connections (Bonderup Dohn, Cranmer, Sime, de Laat and Ryberg, 2018; Goodyear, Carvalho & Bonderup Dohn, 2014) and focus on technologies, communication resources (i.e. sign systems), representations and activities. The technologies and their functions are therefore regarded as important. In addition, the multimodal layers relate to the semiotic properties of technology, how they inhere and prompt sign systems in different ways as interpreted by the actors and are reshaped into modes of representation in different activities. The conclusion is that multimodal methodology, particularly the multimodal layer approach, seems to be beneficial to unpack the relationships and connections between the means used and the actors in NL environments via its coherent approach. A greater understanding of the detailed aspects of technology use in teaching and learning may also be obtained if the existing multimodal layers are accounted for and connected. Insights can guide stakeholders on how to integrate technology in future practices and inform technology choices in relation to specific activities.

To look for another thing, and in another way: revitalising criticality with multimodal methodologies

ABSTRACT This article proposes multimodal methodologies as a path to revitalising criticality in CTS. It begins by assessing the strengths of existing scholarship on “terrorism”, whilst also noting this scholarship’s overwhelmingly linguistic source materials. It considers shortcomings of such a monomodal methodology: including the reproduction of binary conceptual codes, which I argue are uniquely highly codified in linguistic sources; and an imbalance towards structured logics, rather than unstructured affects, in CTS’ conception of terrorism discourses. The article suggests greater diversity of source materials can resolve these shortcomings: deepening our critical project by heightening our sensitivity to transformative, non-binary ways of knowing; and reorienting our analyses from logical to affectual modes of cognition. I demonstrate these possibilities with a review of multimodal materials from Northern Ireland’s so-called “peace walls” (built by state security to contain violence of the “Troubles”). I end with reflections on why CTS might have operationalised a predominantly textual methodology to date – and, further, with a call for readers to integrate multimodality in disseminating our research, as much as selecting our source materials.

Uncovering the emotional dimension of developing teaching practice through uncertainty: a multimodal methodology

ABSTRACT This study explored how academics perceived the development of their practice during emergency remote teaching. It investigated how they approached the rapid upskilling required and what they found helpful and challenging. Interview findings are discussed in relation to a conceptual framework of professional development to reveal shifts in academics’ conceptions when faced with a highly disruptive environment. Findings highlight the importance of capturing non-verbal data to uncover the emotional dimension of teaching development through uncertainty. In addition to assisting reflection on the design of professional development during crises, the study underscores the value of multimodal methodologies in understanding academics’ professional development needs.

Auditory Learner

Auditory Learner

A novel vehicle collision detection system: Integrating audio-visual fusion for enhanced performance

In vehicular accidents, the swift and accurate identification of car crashes is paramount, as it serves as the linchpin for prompt responses from emergency services and search-and-rescue operations. This study introduces an innovative multimodal car crash detection system that capitalizes on audio-visual data sourced from dashboard cameras, thus significantly enhancing the precision of automobile collision detection. In contrast to car crash detection systems relying on single-modal inputs, this research employs advanced multimodal methodologies with a neural network architecture that integrates Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) for car crash detection. This system concurrently leverages audio spectrogram and audio feature data, video RGB, video optical flow, and video differential information, enabling collision recognition through a neural network system, resulting in markedly improved collision detection accuracy. A comprehensive comparative analysis is conducted, benchmarking our system against the most cutting-edge existing car crash detection systems. The system employs a unique fusion approach, allowing it to apply its inference to audio-visual information of varying durations compared to the training set. The empirical findings unequivocally affirm our proposed solution’s substantial enhancement in car crash detection.

Abstract 936: Targeted proteomics and next-generation sequencing (NGS) for biomarker discovery in metastatic colorectal cancer (mCRC)

Abstract Background: Multi-omics analysis is widely used in cancer diagnosis, prognosis and prediction of treatment response. Here we introduce the use of targeted proteomics coupled with NGS as tools to assess oncogenic alterations along with quantitative protein expression in patients with mCRC. Our proteomics panel included proteins that are antibody drug conjugates (ADCs) targets (ERBB 1-4, TROP2, CEACAM5 etc.) as well as proteins involved in sensitivity to ADC payload (SLFN11, drug efflux pump, TOP1 etc.). Methods: The HORIZON III trial (NCT00384176) evaluated FOLFOX + bevacizumab or cediranib as first-line treatment in patients with mCRC (Schmoll J Clin Oncol 2012). In the current analysis, a subset of samples from the HORIZON III trial were assessed for molecular alterations using NGS and proteomics. Pre-existing HER2 IHC scores were used as previously referenced (Cecchi J Clin Oncol 2023). Samples were processed (n=135) with an automated extraction and library preparation method, then sequenced with both a Whole Exome and targeted-custom gene panel. NGS analysis had a success rate of 116/135 samples (87.3%). Absolute quantitation of protein targets was performed using parallel reaction monitoring (PRM) mass spectrometry after collecting tumor epithelial cells by laser microdissection and punch biopsies (n=60) followed by trypsin digestion. Results: The NGS analysis showed that patients had mutated RAS (50%), BRAF (10%), PIK3CA (16%), and TMB-H (18%) and MSI-H (5%) status. A distinguished pattern of mutual exclusivity among KRAS, NRAS, and BRAF mutations is evident. The HER2-positive patients were wildtype RAS, BRAF, and PIK3CA, MSS and TMB-L. Samples that were IHC HER2-positive matched those with NGS ERBB2 amplification. Using PRM, nearly all the evaluable samples had detectable HER2. HER2 IHC 3+ sample showed the highest protein expression at 30,000 amol/ug (equivalent to 1M receptors/cell). Within the HER2-negative patients, HER2 protein expression showed broad dynamic range. CEACAM5 and HER2 are the only ADC targets that reached the level of 1M receptors/cell. Highest TROP2 protein expression is 5 times lower than the highest HER2 value. Conclusions: Our preliminary findings for this cohort indicate the benefit of the use of multi-modal methodologies to inform better strategies for patient stratification and precision medicine, though further assessment of proteogenomic characteristics is needed. This data may inform future clinical development and guide patient eligibility to clinical trials and possible treatment sequencing for ADCs. Citation Format: William J. Kelly, Barrett Nuttall, Marlon Rebelatto, Marcus Schick, Armin Meier, Steve Sweet, Beom-Jun Kim, David Chain, Yeoun Jin Kim, Zhongwu Lai, Ying Wang, Todd H. Creasy, Ramya Gurrapu, Sakshi Gulati, Alice Barkell, Brian Dougherty, Danielle Carroll, Fabiola Cecchi. Targeted proteomics and next-generation sequencing (NGS) for biomarker discovery in metastatic colorectal cancer (mCRC) [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 936.

Advanced Techniques for Geospatial Referencing in Online Media Repositories

In the digital transformation era, video media libraries’ untapped potential is immense, restricted primarily by their non-machine-readable nature and basic search functionalities limited to standard metadata. This study presents a novel multimodal methodology that utilizes advances in artificial intelligence, including neural networks, computer vision, and natural language processing, to extract and geocode geospatial references from videos. Leveraging the geospatial information from videos enables semantic searches, enhances search relevance, and allows for targeted advertising, particularly on mobile platforms. The methodology involves a comprehensive process, including data acquisition from ARD Mediathek, image and text analysis using advanced machine learning models, and audio and subtitle processing with state-of-the-art linguistic models. Despite challenges like model interpretability and the complexity of geospatial data extraction, this study’s findings indicate significant potential for advancing the precision of spatial data analysis within video content, promising to enrich media libraries with more navigable, contextually rich content. This advancement has implications for user engagement, targeted services, and broader urban planning and cultural heritage applications.

Chemometrics-driven monitoring of cheese ripening: a multimodal spectroscopic and scanning electron microscopy investigation.

The integration of spectroscopic techniques with chemometrics offers a means to monitor quality changes in dairy products throughout processing and storage. This study employed Attenuated Total Reflectance-Mid-Infrared Spectroscopy (ATR-MIR) coupled with Independent Components Analysis (ICA), and 3D Front-Face Fluorescence Spectroscopy (FFFS) paired with Common Components and Specific Weight Analysis (CCSWA). The research focused on Cheddar cheeses aged for 1, 2, 3, and 5 years, alongside Comté cheeses aged for 6, 9, and 12 months. The adopted approach offered valuable insights into the intricate cheese aging process within the food matrix. The ICA proportions and CCSWA scores highlighted the significant impact of biochemical transformations during maturation on the aging process. The extracted independent components (ICs) revealed variations in the vibration modes of amides, lipids, amino acids, and organic acids, facilitating the distinction between different cheese age categories. Additionally, CCSWA outcomes identified age-related differences through shifts in tryptophan fluorescence characteristics as the cheeses aged. These results were consistent with the observed alterations in the microstructure of cheese samples over time, corroborated by Scanning Electron Microscopy (SEM) imagery. The introduced multimodal methodology serves as a significant asset for determining the ripening stage of various types of cheese, offering a detailed perspective of cheese maturation beneficial to the dairy industry and researchers.

Exploring MediaPipe optimization strategies for real-time sign language recognition

The present study meticulously investigates optimization strategies for real-time sign language recognition (SLR) employing the MediaPipe framework. We introduce an innovative multi-modal methodology, amalgamating four distinct Long Short-Term Memory (LSTM) models dedicated to processing skeletal coordinates ascertained from the MediaPipe framework. Rigorous evaluations were executed on esteemed sign language datasets. Empirical findings underscore that the multi-modal approach significantly elevates the accuracy of the SLR model while preserving its real-time capabilities. In comparative analyses with prevalent MediaPipe-based models, our multi-modal strategy consistently manifested superior performance metrics. A distinguishing characteristic of this approach is its inherent adaptability, facilitating modifications within the LSTM layers, rendering it apt for a myriad of challenges and data typologies. Integrating the MediaPipe framework with real-time SLR markedly amplifies recognition precision, signifying a pivotal advancement in the discipline.

A novel multimodal approach for the assessment of phlebotomy performance in nurses

Abstract Objectives Phlebotomy is presumably the most challenging preanalytical aspect in laboratory medicine. In Europe, inpatient phlebotomy is performed by nurses in 45–60 % of cases. We aimed to develop and test a novel phlebotomy assessment tool for nurses. Methods A group of 24 nurses working in a surgical ward was investigated. A three-pronged approach was devised: (1) a standardized knowledge test, (2) three blinded phlebotomy audits, and (3) prospective monitoring of samples sent from the investigated surgical ward for the calculation of preanalytical quality indicators. Results The average knowledge test score was 22.7/31 points (12–31, interquartile range 20.5–25). The average audit score was 14.5/18 points (13.7–14.7, interquartile range 14–15). No statistically significant correlations were found between phlebotomy knowledge (or lack of) and corresponding phlebotomy practices (or errors, respectively). Moreover, there was no statistically significant correlation between individual knowledge scores and audit scores. Several misconceptions about the preanalytical phase were identified, along with common phlebotomy errors. Conclusions Sometimes, nurses do not follow guidelines due to lack of theoretical knowledge. Other times, nurses fail to follow procedures despite having the prerequisite theoretical knowledge. We observed a discordance between theory and practice regarding certain aspects of phlebotomy. The novel multimodal methodology presented here describes an improved assessment tool and a superior alternative to the popular survey studies. This tool may be used to identify specific and recurrent phlebotomy issues and to improve institutional continuing education programs for nurses through targeted training programs.

Ethnotheatre and the performative interview as a multimodal methodology of world-making: a case study on young confinement immobility

ABSTRACT Ethnotheatre is an art-based methodology and a critical and world-making practice through reflexive affect endeavour. It joins a group of people to participate in a theatre production about their symbolic and material world. We will debate the effects of this methodology on making ethnography the punctum of socially engaged performance projects. How can we research combining ethnography’s gesture with artistic methodologies to learn common sociocultural riddles or quandaries of marginal and hopefully transitory people? Ethnotheatre effectively elicits the vernacular metacommunicative repertoire veiled in inmates’ hidden transcripts. We explore the framing and potentials of the performative interview in a case study of immobile confinement. Combining ethnographic gesture with artistic world-making practice, we better apprehend and understand the symbolic sites where people may speak out. The world emerges when you make someone heard. We will discuss this methodological tool with a case study about confinement immobility, applying ethnotheatre to understand better how inmates live and resist in their prison’s “invisible” ruled world.

The Toy Brick as a Communicative Device for Amplifying Children’s Voices in Research

The Toy Brick as a Communicative Device for Amplifying Children’s Voices in Research

Chiral and Structural Polymorphism of Fibril Architectures of Homologous Lysozymes.

Amyloid fibrils are fascinating and complex structures with the multilayered chiral organization. Using the multimodal methodology, including VCD, ECD, cryo-EM, and TEM, we characterized in detail different levels of organization (secondary structure/protofilament/mesoscopic structure) of amyloid fibrils prepared from proteins highly homologous in the structure (hen egg white and human lysozymes). Our results demonstrate that small changes in the native protein structure or preparation conditions translate into significant differences in the handedness and architecture of the formed fibrils at various levels of their complexity. In particular, fibrils of hen egg white and human lysozymes obtained in vitro at the same preparation conditions, possess different secondary structure, protofilament twist and ultrastructure. Yet, formed fibrils adopted a relatively similar mesoscopic structure, as observed in high-resolution 3D cryo-EM, scarcely used up to now for fibrils obtained in vitro in denaturing condition. Our results add to other puzzling experiments implicating the indeterministic nature of fibril formation.

Trapping Ecosystems

On June 16, 2018, Beyoncé and Jay-Z released “Apeshit”—a trap-styled hip hop track featuring a chorus of “I can’t believe we made it / Have you ever seen the crowd going apeshit?” The much-commented-on music video for the track was framed as a hip hop takeover of the world’s most visited museum—Paris’s Louvre—featuring pop’s reigning power couple, marketed as “The Carters,” making themselves at home with a collection of dancers in flesh-colored black, brown, and beige bodysuits. While the video was generally received through the split-screen frame of either a cutting decolonial takedown of this monument to Western civilization or the ultimate in money-flaunting bling spectacle, a more subtle and complex set of issues is at play. This article examines the deep historical ambivalences at play in this pop cultural artifact. Employing multi-modal methodologies that combine visual and musical arts perspectives articulated via the frames of postcolonial studies, this analysis theorizes the cultural “traps” in effect. Ranging from the track’s “trap” sonic production and lyrical rhetoric of escape (“we made it”), to the historical trap of musealized colonial plunder and the Louvre’s labyrinthine, oft-subterranean floor plan, to the “trappings” of consumption, bourgeois self-making, and aesthetic contemplation, we seek to illustrate how this socio-cultural text destabilizes Enlightenment universalism and its public/private split.

In Dialogue: Mapping Our Truths—Envisioning the Future of Multimodal Research for Racial Justice

With funding from the American Educational Research Association (AERA), Marva Cappello, Jennifer D. Turner, and Angela M. Wiseman convened a group of critical multimodal scholars in April 2022 to initiate a national agenda that prioritizes the use of visual and multimodal methodologies to promote educational equity and racial justice for youth of color. Our conference gathering included Reka Barton, Darielle Blevins, Justin Coles, Autumn A. Griffin, Stephanie P. Jones, Alicia Rusoja, Amy Stornaiuolo, Claudine Taaffe, Tran Templeton, Vivek Vellanki, and Angie Zapata. The dialogue presented in this article centers around a collaboratively composed image (see ) created three months after our initial convening. Participants from the conference chose an image that reflected our time together and represented our hopes and dreams moving forward. Inspired by kitchen-table talk methodology (), we share our ideas through images and text reflecting on how critical visual and multimodal methodologies facilitate access, equity, and hope in education and educational research.