Multimodal Forms Research Articles

BatLIT Community Extension Project, Bataan, Philippines, in Aid of Contextualization and Enhancement in the Basic Education

BatLIT is an acronym for Bataan Literature-Inspired Teaching, which is a community extension project in support of the Department of Education (DepEd) K-12 program in the Philippines that mandates contextualization and enhancement, also termed ‘Making the Curriculum Relevant to Learners’. The directive calls that in the delivery of instruction, the ‘examples, activities, songs, poems, stories, and illustrations are based on local culture, history, and reality relevant to the learners and easy to understand’. It is therefore aimed that local learning materials and literary outputs in multi-modal forms be produced for distribution to government and private institutions. Moreover, the project is an offshoot of two published research studies on Bataan literature that pioneered the retrieval, documentation, and dissemination of local writings in Bataan, Philippines. Having gathered sufficient volumes of local works, the Bataan Peninsula State University College of Education has pioneered the cascade of research outputs into extension activities to three pilot school beneficiaries representing basic education. The paper utilized the Input-Process-Outputs-Outcome (IPOO) framework as the methodology in the study which paved the way for its completion. The specified objectives of conducting regular monitoring, follow-ups, and visits to partner agencies, providing training workshops for teachers, creating learning materials through multi-modal texts, and integrating Bataan literature in the classroom lessons were attained productively. The results of the project yielded an expansion of the community extension activities to other agencies and educational institutions. Keywords: Community Extension, Bataan literature, Contextualization and Enhancement, Humanities, IPOO framework

Fully foldable mechanical metamaterials with isotropic auxeticity and its generated multi-mode folding form

Abstract Auxetic materials, a type of mechanical metamaterial with negative Poisson's ratios, are potentially utilized in the realms of energy absorption and engineering structures. However, most of the existing auxetic materials either contain a large amount of rotational motion or still have gaps when fully folded, which is not conducive to lifting loads. Besides, their application is limited to flexible environments due to their single-folding mode. To overcome such limitations, a fully foldable mechanical metamaterial with isotropic auxeticity is proposed by utilizing the Sarrus mechanism, and its generated multi-mode folding form is obtained in this paper. Then, the DOF, bistability and kinematic characterizations are analyzed to show the performance of the proposed structures. Finally, the parameters of the proposed fully foldable mechanical metamaterials are discussed to simplify the structures. Some prototypes are fabricated to validate the effectiveness and performance of the proposed mechanical metamaterials. The proposed mechanical metamaterials have some merits, such as isotropic auxeticity, being fully folded to achieve dense compression, being bistable with load-bearing capacity, multi-mode folding form, and one-DOF, and they have versatile potential applications in complex environments requiring high-impact resistance and flexible adaptation.

A prediction model for the mechanical properties of SUS316 stainless steel ultrathin strip driven by multimodal data mixing

Constructing a mapping relationship among material preparation process, microstructure, and mechanical properties is a challenge in material research and development. In this work, a deep learning framework for multimodal data fusion is constructed that couples a multi-layer perceptron (MLP) and a residual neural network (ResNet) to predict mechanical properties of SUS316 stainless steel ultrathin strips. Specifically, the MLP branch is used to extract the rolling process data features, and the ResNet with the addition of a convolutional block attention module (CBAM) is used to extract the microstructure features. Six models are constructed for comparison under the comprehensive consideration of factors such as unimodal network, the multimodal network and input form of image samples. The results show that the multimodal data model fused with the ResNet and MLP after adding the CBAM using both rolling process data and four types of microstructure image data as model inputs has the most accurate prediction results. The R2, MAPE, RMSE and MAE are 0.998, 0.727, 4.440 and 3.359, respectively. In addition, the proposed model is used for predicting yield strength and elongation, and the results show that the R2 values of both models on the test set are greater than 0.980, fully confirming that the multimodal data model has high prediction accuracy and good generalizability.

Automatic Modulation Recognition Based on Multimodal Information Processing: A New Approach and Application

Automatic modulation recognition (AMR) has wide applications in the fields of wireless communications, radar systems, and intelligent sensor networks. The existing deep learning-based modulation recognition models often focus on temporal features while overlooking the interrelations and spatio-temporal relationships among different types of signals. To overcome these limitations, a hybrid neural network based on a multimodal parallel structure, called the multimodal parallel hybrid neural network (MPHNN), is proposed to improve the recognition accuracy. The algorithm first preprocesses the data by parallelly processing the multimodal forms of the modulated signals before inputting them into the network. Subsequently, by combining Convolutional Neural Networks (CNN) and Bidirectional Gated Recurrent Unit (Bi-GRU) models, the CNN is used to extract spatial features of the received signals, while the Bi-GRU transmits previous state information of the time series to the current state to capture temporal features. Finally, the Convolutional Block Attention Module (CBAM) and Multi-Head Self-Attention (MHSA) are introduced as two attention mechanisms to handle the temporal and spatial correlations of the signals through an attention fusion mechanism, achieving the calibration of the signal feature maps. The effectiveness of this method is validated using various datasets, with the experimental results demonstrating that the proposed approach can fully utilize the information of multimodal signals. The experimental results show that the recognition accuracy of MPHNN on multiple datasets reaches 93.1%, and it has lower computational complexity and fewer parameters than other models.

Reading comics: The effect of expertise on eye movements

The theory of expertise suggests that there should be observable differences in the eye movement patterns between experts and non-experts. Previous studies have investigated how expertise influences eye movement patterns during cognitive tasks like reading. However, the impact of expertise on eye movements in comics, a multimodal form of text, remains unexplored. This article reports on a study that uses eye tracking to examine patterns in the ways that experts and non-experts read comics. Expert participants (14) with experience in reading comics than non-expert participants (17). When controlling for variables such as layout and text quantity, we found significant differences in visual scanning between experts and non-experts. Experts exhibited more frequent saccades and greater amplitude of saccades. Further analysis revealed distinct strategies in processing text and image content between the two groups: the interaction between expertise level and content type in specific AOI showed significant differences across multiple visual measurement metrics, including Average duration of fixations, number of fixations, and number of saccades within AOI. These findings not only support the applicability of the expertise level theory in the field of comic reading but also provide a new perspective for understanding the reading processing of multimodal texts.

Expressive language in translation of books for children: a corpus-based study

The paper focuses on translation solutions used in translating expressive language from English into German, Croatian and Serbian. This qualitative analysis is corpus-based and it includes translations of a children’s book from the series Diary of a Wimpy Kid by Jeff Kinney. These books are written for children aged 9 ‒12 and represent a kind of a hybrid multimodal form because they contain text accompanied by illustrations in the form of a comic with speech balloons, sound effects and similar. By expressive language we mean wordplay, onomatopoeic words and funny names, frequently employed in children’s literature. The source text is in English and the analysis will be based on solutions and techniques used in translations of this bestseller in Croatian, German and Serbian. The main aim of this research is to determine the functions of expressive language and the solutions translators used in order to achieve the same or similar effect in the target languages. The results of the study will give a better insight into the role of the translator in the whole process and the ways expressive language is translated. Keywords: expressive language; translation solutions; multimodal discourse; children’s literature

The Effects of Intra-Operative Lidocaine Infusion on Post Operative Pain and Morphine Consumption Following Major Gynaecological Surgeries Under General Anaesthesia

<i>Introduction</i>: Major gynaecological surgeries are associated with considerable postoperative pain, which remains a challenge for many practitioners. Multimodal forms of analgesia significantly reduce the requirement of opioids for pain management. Despite its local anaesthetic effects, lidocaine infusion improves postoperative pain and morphine consumption following gynaecological surgeries. <i>Materials and methods</i>: Sixty patients were assigned randomly into 2 groups (A and B) with 30 patients per group. Group A received intravenous lidocaine 1.5 mg/kg at induction via a bolus injection and 1.5 mg/kg/hr in normal saline infusion from onset of surgery to the end of surgery, while the control group (Group B) received equal volume of normal saline at the same timelines. Pain scores were assessed postoperatively using the numerical rating scale and the cumulative morphine consumed postoperatively were also measured. <i>Results</i>: The mean pain scores were significantly higher in the Saline Group than in the Lidocaine group. The cumulative morphine consumption after 48 hours was significantly reduced in the study group 4.87 ± 1.80 mg vs 14.13 ± 4.10 mg (P<0.0001). Conclusion: The administration of a bolus dose (1.5 mg/kg) of intravenous lidocaine at induction and a continuous intravenous infusion of 1.5 mg/kg/hr from onset of surgery till skin closure reduced the postoperative pain intensity and morphine consumption in patients undergoing major gynaecological surgeries under general anaesthesia.

Audiovisual Imagery in Multimodal Perception and Performance of Qigang Chen’s Er Huang Concerto

In the realm of musical performance, the role of a pianist extends beyond the mere execution of musical scores to become a pivotal contributor to artistic expression. This study delves into the auditory and visual imagination through the perception and performance of piano works by the Chinese-French composer Qigang Chen from the early 21st century, involving the performer’s meticulous response to the nuances of musical sound as well as the interplay of visual synaesthesia.The study also explores the inherent conveyance chain within the performance process, experimenting with multimodal forms of expression. These creative choices not only enrich the narrative of the arts but also contribute to crafting a comprehensive, immersive concert experience for the audience. The principal aim of this research is to explore the prismatic view within Qigang Chen’s piano work, offering multiple possibilities of meaning and emphasizing the crucial role of the pianist in forming and elucidating the fusion of auditory and visual senses in performative action.By transcending traditional performance boundaries, this exploration invites the audience to witness the multimodal interactions inherent in musical performances, providing a fresh perspective on dynamic exchanges between different sensory elements and cultural influences.

Emergent bilingual preschoolers’ verbal and embodied engagement behaviors in read aloud

Shared reading is a crucial site for children’s emerging reading skills when children engage affectively, behaviorally, and cognitively in the reading process. To inform a more holistic, collective, and inclusive view of observable engagement in read aloud (RA) behaviors, authors examined EB preschoolers’ micro-moment engagement behaviors to understand how they recruit their verbal and embodied modes to attend to RAs and how these behaviors align with prior engagement typologies. A phenomenological approach was implemented to examine EB preschool students’ moment-to-moment, multimodal forms of engagement across three video recorded RAs that consisted of 258 units of analysis. Findings reveal that EB preschoolers’ attentiveness and attention divergence during RA activities were showcased across a multimodal continuum but did not necessarily indicate disengagement or inattention. Opportunities for children’s contemplation were constrained by social and behavioral expectations of their compliance, and within less dialogic RAs, these expectations undermined children’s multimodal means of textual engagement in favor of their quiet, undivided display of attention. Findings have implications for recognizing the role of young children’s multimodal expression in supporting their engagement and interactions in RA in ways that may contribute to their developing reading skills and outcomes.

Encounter

Anthropologists often talk about ‘encounters,’ but what do they actually mean? This term—‘encounter’—shows up everywhere across ethnographic writing and practice, but is itself rarely defined or discussed. ‘Ethnographies of encounters,’ too, are increasingly common, but are rarely treated as a distinct type of ethnography. In this short essay, I recommend approaching the concept of encountering more consciously, but also defining it in a relatively basic and expansive way. Encounters across difference are open-ended: they can lead to collaboration, conflict, negotiation or an awkward disconnect, to name a few. Encounters are many-sided: much beyond a single studied group of people, beyond the ethnographer’s home society, and more than human. Assuming that encounters are very open-ended and many-sided brings new methodological and ethical challenges. Yet, it helps expand our attention and care in fieldwork and analysis, while postponing our judgements. Keywords: ethnographies of encounter, alterity, relationality, world-making, definitions This essay belongs to the essay series “The Anthropologist’s Toolkit: Reflections on ethnographic methodology”. In this series, authors peer into the anthropologist’s toolkit to reflect on what ethnographic methodology constitutes in all its multimodal forms.

Interactive design of new media images and architectural space: taking Hangzhou as an example

ABSTRACT Traditional architectural space design often relies on fixed display boards, paper guides, and static displays. Viewers can only passively receive information and lack opportunities to interact with the displayed content, making it difficult to generate deep participation and personalized experiences. This article applied new media images into the interactive design of architectural spaces, transforming and innovating traditional architectural spaces, and providing audiences with a richer and more attractive spatial experience. The navigation service was set as the architectural space design goal, and analyzed using the Leifeng Tower building in Hangzhou as an example. Building Information Modeling (BIM) was used to digitally model the Leifeng Tower building, accurately simulating the building structure, spatial layout, and landscape information. By adding virtual information annotations and navigation routes, tourists were provided with a more diverse and colorful architectural space experience. The experimental results showed that among 400 tourists, the highest number of tourists using augmented reality navigation was 204, and the average satisfaction score of tourists with augmented reality navigation reached 90. Moreover, presenting the guide content in a multimodal form can better enhance the spatial experience of tourists. The use of augmented reality in architectural spatial interaction design can enrich tourists’ perception and experience in the architectural space, enabling them to interact with the building in a more vivid and interactive way.

Patient and quality characteristics in the treatment with disulfiram (Antabus) in the German "Network for Alcohol Aversive Pharmacotherapy"

More than adecade ago disulfiram lost its approval for use in Germany. Nonetheless, aconsiderable number of psychiatric hospital outpatient departments as well as practicing physicians continue to prescribe it. These professionals have formed the "Network for Alcohol Aversive Pharmacotherapy" (NAP) to maintain a high quality of this treatment approach. To describe the current use of disulfiram with respect to patient numbers and characteristics, side effects, and use of concomitant multimodal treatment forms. Since 2019 the NAP has conducted an annual retrospective survey among its members regarding the aforementioned parameters. From 2019 to 2023 atotal of 1579 treatment cases were described by 33centers, 152 patients reported atotal of 241 drinking events, 26of them resulting in hospitalization but none causing complications or permanent harm. The most frequent side effects, in descending order, were unpleasant body odor (2.5%), fatigue, male sexual dysfunction, mildly elevated liver enzymes, allergic skin reactions and polyneuropathy (0.8%). More than one quarter of the patients suffered from comorbid depression, and approximately 5% from ADHD, borderline or other personality disorders, trauma-related disorders and anxiety disorders, respectively. Of the patients 33% were treated with antidepressants and 12% with sedating antipsychotics. Various forms of concomitant group therapy were offered to 66% of the patients. Treatment with disulfiram is legally possible, generally well-tolerated and safe. It is offered in most treatment centers as part of acomprehensive treatment plan that includes multimodal treatment of comorbid psychiatric disorders.

Effects of Multimodal Physical Therapy on Pain, Disability, H-reflex, and Diffusion Tensor Imaging Parameters in Patients With Lumbosacral Radiculopathy Due to Lumbar Disc Herniation: A Preliminary Trial.

Background Lumbosacral radiculopathy (LSR) due to lumbar disc herniation (LDH) is a condition caused by mechanical compression of nerve roots. Various physical therapy interventions have been proposed for the conservative management of LSR due to LDH. However, the study of physical therapy interventions in a multimodal form is lacking. Additionally, the effect of physical therapy on diffusion tensor imaging (DTI) parameters of the compressed nerve root has not been studied. This study aimed to investigate the effects of multimodal physical therapy (MPT) on pain, disability, soleus H-reflex, and DTI parameters of the compressed nerve root in patients with chronic unilateral LSR due to LDH. Methods A prospective preliminary pre-postclinical trial with a convenience sample was conducted. A total of 14 patients with chronic unilateral LSR due to paracentral L4-L5 or L5-S1 LDH were recruited for the study. Participants received a total of 18 sessions of a six-weekMPT program that consisted of electrophysical agents, manual therapy interventions, and core stability exercises. Electrophysical agents involved interferential current and hot pack. Manual therapy interventions included myofascial release, side posture positional distraction, passive spinal rotation mobilization, and high-velocity low-amplitude manipulation. Visual analog scale (VAS), Roland-Morris Disability Questionnaire (RMDQ), soleus H-reflex amplitude, side-to-side amplitude (H/H) ratio, fractional anisotropy (FA), and apparent diffusion coefficient (ADC) of the compressed nerve root were measured at baseline and post-intervention. Results There were significant improvements in VAS, RMDQ, H/H ratio, FA, and ADC of the compressed nerve root. Furthermore,significant improvement was found in the affected side compared with the contralateral side in H-reflex amplitude. Conclusions The observations of this preliminary trial suggest that MPT is a successful intervention in patients with chronic unilateral LSR due to LDH. Regarding DTI parameters of the compressed nerve root, FA increased and ADC decreased. Future studies with a control group, large sample sizes, and longer follow-up periods are needed.

포스트휴먼 시대의 하이퍼텍스트의 미학적 윤리

This paper aims to search for the aesthetical ethics through understanding the multimodal text, Brian Selznick’s The Invention of Hugo Cabret, and to suggest the vital vision for the survival of literature at the digital culture in the posthuman age. As an orphaned boy, Hugo finds the history of silent movie and recovers George Méliès’s identity as the silent movie producer. His investigation into the past turns out to find the newness and to open the vibrant future. As this hybrid graphic novel shows the changes from the written text to digital hypertext, it asserts that new media is constructed on the basis of old previous media, and we should find the solutions in the relationships with past and future. This novel resurrects vibrantly the multimodal form borrowed from the cinematic technique through the printed book. As the genres are mixed at the process of creating the new media, the complex combinations with image and text represent the microsome of media environment. Selznick’s vital multimodal form initiates an innovative storytelling, because it gives the opportunities to find the futuristic way in the excessive digital culture and to achieve the ethical vision through the participatory reading.

The cognitive roots of multimodal symbolic forms with an analysis of multimodality in movies

Condillac’s (1754) “Traité des sensations” is the philosophical background of modern discussions on the relationship between perception and multimodal communication. The differences between perception and communication and the transitions between them are discussed with a focus on odor and color. It becomes clear that even at this primary level, the complex interactions of different modalities are the precondition for effective and rich communication. The second part discusses Cassirer’s “Philosophy of Symbolic Forms” as a relevant framework for multimodality studies. Basic aspects are first commented on with a focus on music and visual art. The interaction is even more complex and rich in the case of language; the difficulty of large symbolic forms is mainly due to semantic composition and only to a lesser degree to syntactic concatenation. The first must merge/blend different semantic spaces. It must allow for the plurality of levels of integration from the lexical level, the level of phrases and sentences, up to texts and discourse. The third part focuses on multimodality in film. It treats the representation of movement and action in (film) narratives, the visual perception and representation/communication of movement and action, and the integration of music, moving images, and language.

Reflections on Ethnographic Re-enactment

"The Anthropologist's Toolkit: Reflections on ethnographic methodology" is a new essay series that peers into the anthropologist’s toolkit to reflect on what ethnographic methodology constitutes in all its multimodal forms. We invite all contributions that explore the multiple engagements that come into play when working ethnographically with humans and more-than-humans.

Your body tells how you engage in collaboration: Machine‐detected body movements as indicators of engagement in collaborative math knowledge building

AbstractCollaborative learning, driven by knowledge co‐construction and meaning negotiation, is a pivotal aspect of educational contexts. While gesture's importance in conveying shared meaning is recognized, its role in collaborative group settings remains understudied. This gap hinders accurate and equitable assessment and instruction, particularly for linguistically diverse students. Advancements in multimodal learning analytics, leveraging sensor technologies, offer innovative solutions for capturing and analysing body movements. This study employs these novel approaches to demonstrate how learners' machine‐detected body movements during the learning process relate to their verbal and nonverbal contributions to the co‐construction of embodied math knowledge. These findings substantiate the feasibility of utilizing learners' machine‐detected body movements as a valid indicator for inferring their engagement with the collaborative knowledge construction process. In addition, we empirically validate that these inferred different levels of learner engagement indeed impact the desired learning outcomes of the intervention. This study contributes to our scientific understanding of multimodal approaches to knowledge expression and assessment in learning, teaching, and collaboration.Practitioner notesWhat is already known about this topic Previous research emphasizes the importance of gestures as essential tools for constructing common ground and reflecting shared meaning‐making in learning and teaching contexts. The prior studies in multimodal learning analytics (MMLA) suggest that certain forms of body movements and postures can be differentiated based on the automatic detection of upper body joint locations. Empirical observations indicate that co‐thought gestures typically involve smaller hand or arms movement that are closer to the gesturer's body than co‐speech gestures used in interpersonal communication. What this paper adds This paper fills the research gap by examining the use of gestures in collaborative learning, offering insights into how individuals contribute verbally and nonverbally to collaborative knowledge construction. This paper introduces the concept of using machine‐detected body movements as a viable proxy for inferring learners' engagement in collaborative knowledge‐building activities. Leverages sensor technologies for automatic detection of body movements, the innovative approach in this work seeks to overcome the time‐intensive and laborious process of manually coding gestures. Implications for practice and/or policy By recognizing the potential significance of learners' body movements in indicating engagement levels with collaborative knowledge‐building activities, instructors can set up computer‐supported collaborative learning (CSCL) environments to enable capturing these movements. Given the crucial role of gestures in learning, teaching, and collaboration, educators can create more equitable formative assessment practices for linguistically diverse students by developing strategies that align with multimodal forms of knowledge expression. Research can expand beyond mathematics to explore the transferability of these findings to other subjects, helping educators create comprehensive pedagogical approaches that leverage multimodal interactions across disciplines.

Fuser: An enhanced multimodal fusion framework with congruent reinforced perceptron for hateful memes detection

As a multimodal form of hate speech on social media, hateful memes are more aggressive and cryptic threats to the real life of humans. Automatic detection of hateful memes is crucial, but the images and texts in most memes are only weakly consistent or even irrelevant. Although existing works have achieved the initial goal of detecting hateful memes with pre-trained models, they are limited to monolithic inference methods while ignoring the semantic differences between multimodal representations. To strengthen the comprehension and reasoning of the hidden meaning behind the memes by combining real-world knowledge, we propose an enhanced multimodal fusion framework with congruent reinforced perceptron for hateful memes detection. Inspired by the human cognitive mechanism, we first divide the extracted multisource representations into main semantics and auxiliary contexts based on their strength and relevance, and then precode them into lightly correlated embeddings with unified spatial dimensions via a novel prefix uniform layer, respectively. To jointly learn the intrinsic correlation between primary and secondary semantics, a congruent reinforced perceptron with brain-like perceptual integration is designed to seamlessly fuse multimodal representations in a shared latent space while maintaining the feature integrity in the sub-fusion space, thereby implicitly reasoning about the subtle metaphors behind the memes. Extensive experiments on four benchmark datasets fully demonstrate the effectiveness and superiority of our architecture compared with previous state-of-the-art methods.

A Review of Clinical Practice Guidelines for Physiotherapy Management of Patients with Non-Specific Low Back Pain

Background: Despite the availability of various clinical practice guidelines (CPGs) offering evidence-based recommendations, there exists a global diversity in physiotherapy practices for managing patients with non-specific low back pain (NSLBP). In this review, we aim to critically appraise and map the current evidence regarding physiotherapy management for patients with NSLBP. Methods: A search strategy was formulated and searched in MEDLINE, Scopus, EBSCOhost, PEDro and Google Scholar between 2013 and 2023 in English. Critical appraisal was performed using International Centre for Allied Health Evidence (iCAHE) guideline checklist. Results: A total of 12 CPGs met the selection criteria were synthesised. Exercise therapy, physical activity and manual therapy were consistently recommended for patients with NSLBP across most guidelines, irrespective of symptom duration. Cognitive-behavioural therapy was frequently advised for those with chronic NSLBP. However, notable absences of recommendations for electrotherapy, mechanical traction, bed rest, and lumbar corsets in NSLBP management. Moreover, emerging treatments such as shock wave therapy and cognitive-functional therapy were not extensively addressed in the guidelines. Conclusion: Multimodal and active forms of physiotherapy management were predominantly recommended in most of the guidelines. However, there is a clear necessity for updated CPG that effectively integrate biopsychosocial approaches into the management of patients with NSLBP.

“Through the looking glass: envisioning new library technologies” AI and the information interface

PurposeThis study aims to inform and guide information professionals in thinking clearly about the challenges and opportunities this technology may create.Design/methodology/approachThis column uses the Web browser Arc as a focal point for exploring elements that seem key to understanding how artificial intelligence (AI) may change our relationship with information. Large language model’s were used to help draft or rewrite sentences. That text was then reviewed or revised by the author.FindingsThe following elements are key to understanding the potential of informational interface software like Arc. The ability to abstract information from the original content. The ability to produce multimedia compelling user experiences. The ability to “read” multimodal forms of information and take action based on that “understanding”. This may impact the value exchange between the user and the underlying information, with implications for libraries.Research limitations/implicationsEverything about AI the future of AI or any technology is speculative.Practical implicationsLibraries that wish to continue to be part of adding value to how users interact with information need to pay attention and find ways to adapt.Originality/valueAs new paradigms are created to ensure information exchange is sustainable for everyone, there may be opportunities for libraries. And even if not, libraries may leverage their expertise or relationships to build something that could not be imagined without them. Yet these are only possible if libraries engage.