Multimodal Presentations Research Articles

Plasticity in older infants' perception of phonetic contrasts: The role of selective attention in context.

Developmental plasticity refers to conditions and circumstances that increase phenotypic variability. In infancy, plasticity expands and contracts depending on domains of functioning, developmental history, and timing. In terms of language processing, infants attend to and discriminate both native and non-native phonetic contrasts, but selectively attune to their native phonemes by the end of the first postnatal year. However, relevant studies have excluded factors regarded as promoters of attention such as infant-directed (ID) speech, synchronous multimodal presentations, and female speakers. Here we investigated whether English-learning 11-month-olds would discriminate a non-native phonetic contrast while manipulating these factors. Results showed significant discrimination of the non-native contrast, regardless of speech register, provided that they were presented by a dynamic female speaker. Interestingly, when a static object or a dynamic male ID speaker replaced the female, no significant discrimination was found. These results show infants to be capable of discriminating non-native phonetic contrasts in an enhanced context at an age when they have been characterized as not being able to do so. Synchronized, multimodal information from female speakers allowed infants to perceive difficult non-native phonemes, highlighting the importance of an ecologically valid context for studying speech perception and language learning in early development.

Examining the Effects of Multimodal Presentations on Learning Spatial Layouts

Presenting information appropriately in multiple modalities (e.g., both text and narration) has previously been shown to positively impact learning in STEM areas. While this effect has proven robust for learning science topics, it has not been explored relative to the formation of mental representations of physical spaces or layouts. In this experiment, participants were given a description of a spatial location, which contained 13 explicit locations or features. Participants either read this description, heard a narrated reading of this description, or instead received both text and narration simultaneously. Recall of the physical space was evaluated using both a multiple-choice test and a map recall task. Results indicated that text alone was always the worst performer, and that presentations that capitalized on the auditory channel led to significantly better learning in both measures. These results thus partially replicate previous research on multimodal learning and extend this research to the novel context of learning navigable space.

Exploring the effects of robot-assisted multimodal composition on students’ audience awareness for English writing

Previous studies have focused on the effects of employing robot-assisted multimodal composition to facilitate learners’ development of audience awareness; however, how the process promotes such awareness is under-explored. In this study, therefore, how a robot-assisted multimodal composition approach might facilitate EFL students’ audience awareness for English writing was investigated. A total of 53 advanced EFL learners at a university of science and technology in Taiwan were engaged in creating multimodal presentations using social robots with the capacity to provide authentic meaning-making environments for students to flexibly design and produce video presentations drawing on various semiotic modes for different purposes and audiences. The data included participants’ pre- and post-intervention English essays, recorded interviews, concept maps, planning worksheets, final video presentations, and reflective surveys. The pre- and post-intervention English essays were analyzed employing a paired-sampled t test, and a multimodal analysis was applied to the other data. The findings revealed that the students’ audience awareness in their English writing improved as they progressed through the robot-assisted processes of identifying their target audiences’ needs and expectations, establishing logical connections with them, designing interactive features with multimodal materials to stimulate audience responses, and constructing effective communication flows so as to create a presentation to be published on YouTube. These processes contributed to the students’ increased sensitivity towards their audiences. Pedagogical and further research implications are discussed.

Three Taiwanese elementary school English teachers’ pedagogical and collaborative practice of glocalization in CLIL lessons

ABSTRACT This case study explored three Taiwanese elementary school English teachers’ pedagogical and collaborative practice of glocalization in content and language integrated learning (CLIL) lessons. Based on the framework on English teachers’ designs and implementing glocalized CLIL lessons, the thematic analysis of the documents, observations of classroom practices and post-observation conferences, and interviews reached the following conclusion. Firstly, food was the major glocalized CLIL topic. Secondly, these three English teachers had positive attitudes towards their multimodal presentations of glocalized CLIL lessons. Thirdly, these three teachers possessed different levels of English language skills in their collaborative lesson planning. Fourthly, these teachers faced challenges in providing their learners with authentic tasks for pushed output (learners’ production of the language of learning and content knowledge for contextualized and meaningful use) and using English as the medium of instruction.

Effects of input modality on alignment in continuation writing

Previous studies demonstrated that alignment entailed in the continuation task can facilitate language learning (e.g., Wang & Wang, 2015), but the investigation into how to maximize alignment to tap greater learning potential is still in its infancy. The present study extends this line of inquiry by examining whether interaction with multimodal presentations would bring about enhanced alignment effect in the reading-writing continuation task. Participants were three groups of Chinese senior English as a foreign language (EFL) learners who were exposed to each of the three input modalities: text modality, text+picture modality and text+video modality. Results showed that 1) text+video modality brought about stronger alignment at both linguistic and situational levels, more accurate and cohesive writing production; 2) text+picture modality only magnified alignment at situation models; 3) learners varied their perceptions towards the incorporation of pictorial and video input. Pedagogical implications of the study were thereafter discussed.

De-mystifying the nimbus of research: re-igniting practitioners' interest in exploring EAP

Engagement in research is recognised as a transformative force for professional development in EAP. Consequently, for many EAP practitioners research and/or scholarship have been added as contractual requirements. However, many EAP professionals desist from engaging in this, as they wonder how to fit it into their workload. In this paper I consider a form of practitioner research, Exploratory Practice, which integrates research and pedagogy in sustainable investigations into practice. I discuss ways in which different EAP practitioners, in a range of international contexts, explore puzzles relevant to their professional development, and disseminate their understandings. Drawing on published studies and vignettes from EAP practitioners enacting Exploratory Practice, I examine the ways in which these activities dispel the nimbus surrounding traditional views of research. This includes conducting robust investigations, and providing multimodal presentations in the supportive atmosphere of BALEAP PIMs and Conferences. In doing so, practitioners enhance their understandings of pedagogy and contribute to the field as equally-valued theorisers, and meaning-makers in EAP.

Craft is ubiquitous

This Special Issue presents a selection of contributions that seek to extend the idea of what craft practice and research can be. They stem from the conference presentations in the 1st Biennial International Conference for the Craft Sciences (BICCS), held online during 4–6 May 2021. This conference was initiated by the Craft Laboratory in Mariestad city, which is affiliated with the Department of Conservation, University of Gothenburg, Sweden. What counts as craft, and what does not, has been discussed with the general consensus that craft often evades definitions and instead thrives as an adhesive between other domains. In this editorial we claim that craft practice is ubiquitous, since acts of ‘crafting’ are infiltrated in most aspects of society, from the industrial workplace to the home. In addition to being a professional domain, craftmanship is also an attitude and a way of life. Craft making further facilitates shared reflective platforms which can carry and sustain cultural associations, or even social resistance, over time. We hope to invite readers to extend the notions of what crafts can be, by discussing issues related to such various topics as plant propagation, crystal growing, neuroscientific activity tracking, multimodal presentations of craft research and hybrid forms of digital and handmade craft processes. We also present an overview of educational contexts of crafts and discuss the role of the craft practitioner in heritage studies such as traditional boat building or industrial lace making.

Multimodal Digital Storytelling Presentations among Middle-School Learners of English as a Foreign Language: Emotions, Grit and Perceptions

Despite the increasing attention to technology-enhanced language learning in English-as-a-foreign-language contexts, investigation regarding the effects of multimodal technologies on affective factors (particularly emotions and grit) in digital storytelling has remained underexplored. Therefore, this mixed-methods study reports on positive/negative emotions, grit and learner perceptions in the digital storytelling presentation process as the result of different presentation modes (robot-assisted versus PowerPoint-assisted). With 52 9th-grade middle-school students from two intact classes in a junior high school in central Taiwan, the results from multiple data sources (an emotion questionnaire, a grit survey, a perception survey, student in-class sharing) revealed that the robot-assisted mode was more advantageous in contributing to more positive emotions and in making learners grittier, foregrounding higher perseverance of effort in the learning process. The students having the robot-assisted mode also responded more positively to the overall learning experience.

Multisensory integration of musical emotion perception in singing

We investigated how visual and auditory information contributes to emotion communication during singing. Classically trained singers applied two different facial expressions (expressive/suppressed) to pieces from their song and opera repertoire. Recordings of the singers were evaluated by laypersons or experts, presented to them in three different modes: auditory, visual, and audio–visual. A manipulation check confirmed that the singers succeeded in manipulating the face while keeping the sound highly expressive. Analyses focused on whether the visual difference or the auditory concordance between the two versions determined perception of the audio–visual stimuli. When evaluating expressive intensity or emotional content a clear effect of visual dominance showed. Experts made more use of the visual cues than laypersons. Consistency measures between uni-modal and multimodal presentations did not explain the visual dominance. The evaluation of seriousness was applied as a control. The uni-modal stimuli were rated as expected, but multisensory evaluations converged without visual dominance. Our study demonstrates that long-term knowledge and task context affect multisensory integration. Even though singers’ orofacial movements are dominated by sound production, their facial expressions can communicate emotions composed into the music, and observes do not rely on audio information instead. Studies such as ours are important to understand multisensory integration in applied settings.

Human mirror neuron system responsivity to unimodal and multimodal presentations of action.

This study aims to clarify unresolved questions from two earlier studies by McGarry et al. Exp Brain Res 218(4): 527-538, 2012 and Kaplan and Iacoboni Cogn Process 8: 103-113, 2007 on human mirror neuron system (hMNS) responsivity to multimodal presentations of actions. These questions are: (1) whether the two frontal areas originally identified by Kaplan and Iacoboni (ventral premotor cortex [vPMC] and inferior frontal gyrus [IFG]) are both part of the hMNS (i.e., do they respond to execution as well as observation), (2) whether both areas yield effects of biologicalness (biological, control) and modality (audio, visual, audiovisual), and (3) whether the vPMC is preferentially responsive to multimodal input. To resolve these questions about the hMNS, we replicated and extended McGarry et al.'s electroencephalography (EEG) study, while incorporating advanced source localization methods. Participants were asked to execute movements (ripping paper) as well as observe those movements across the same three modalities (audio, visual, and audiovisual), all while 64-channel EEG data was recorded. Two frontal sources consistent with those identified in prior studies showed mu event-related desynchronization (mu-ERD) under execution and observation conditions. These sources also showed a greater response to biological movement than to control stimuli as well as a distinct visual advantage, with greater responsivity to visual and audiovisual compared to audio conditions. Exploratory analyses of mu-ERD in the vPMC under visual and audiovisual observation conditions suggests that the hMNS tracks the magnitude of visual movement over time.

Integrating Facebook as a Web 2.0 Tool in a Responsive Pedagogy for Multimodal Oral Presentation Skills

Technology both complicates and liberates oral presentation pedagogy in higher education. On one hand, 21st century communication realities require students to deliver multimodal oral presentations but educators have only begun to experiment with the relevant pedagogy. On the other hand, Web 2.0 offers vast pedagogical potential which practitioners have just begun to explore. Confronted by a pedagogical predicament and intrigued by the affordances of Web 2.0, we designed the Responsive Multimodal Oral Presentation Pedagogy (RMO2P) to enhance the learning of multimodal oral presentation skills among EAP (English for Academic Purposes) students. RMO2P integrated Facebook as a Web 2.0 tool with videos, collaborative learning and feedback. A teacher-researcher implemented RMO2P in a public speaking class of 20 EAP students for a semester. Thematic analysis of multiple sources of data which included student artifacts, observational data and inquiry data contributed to the findings. The findings indicated that Facebook was a feasible Web 2.0 tool that can be effectively integrated with other pedagogical techniques. As a result, the participants demonstrated enhanced conceptual knowledge and heightened self-awareness as presenters of multimodal oral presentations. Since Facebook was integrated as a Web 2.0 tool in a responsive method for multimodal oral presentation skills to overcome situational constraints including inadequate learning management systems (LMS), stakeholders who are seeking alternative methods to enhance the teaching of oral presentation skills may adapt the intervention to enhance learning outcomes. Keywords Facebook; multimodal oral presentation skills; pedagogy; higher education; Web 2.0

“That’s the Wonder of It”: Affective Dimensions of Visual Rhetoric for Biodiversity Conservation

In environmental communication, audience engagement is an essential prerequisite for achieving persuasive aims. This article responds to recent interest in visual storytelling and emotionalization – purposeful display and elicitation of emotions – as engagement techniques. A case study of the 2020 Global Biodiversity Festival – part online science festival, part fundraising event – provides evidence of how these techniques are employed in environmental communication for biodiversity conservation. Informed by scholarship on affect, emotion, visual rhetoric, and environmental communication, the case study analysis shows how visual representations of nature, mediated experiences of nature, and accompanying narration orient festival audiences toward specific ways of seeing and feeling that foreground emotional commitments and draw audiences into potentially transformative encounters. The visual rhetoric and affective dimensions of the festival’s website, virtual fi eld trips, and multimodal presentations focus attention, create moments of connection, and call audiences to action. The case study analysis also reveals how the festival, planned in response to the COVID-19 pandemic, treats this crisis as a kairotic moment for encouraging awareness, care, and pro-environmental behaviors.

Knowledge and perceived ability to deal with physical comorbidities in substance use disorders: Does participation in continuing medical education help?

Background: Despite the magnitude and impact of co-occurring physical and substance use disorders (SUDs) and evidence of effectiveness of integrated care, the training for management of comorbidities has been neglected. Our objective was to examine the effectiveness of 1-day continuing medical education (CME) in this area on participants' perceived and objective knowledge and skills. Materials and Methods: It was a cross-sectional study of CMEs conducted in a tertiary care center with a pre-post assessment design. Assessment included rating perceived knowledge and skill on a five-point scale and objective multiple-choice questions. Subjective feedback on qualitative aspects of the CME was taken. Out of total 154 participants, 69 completed pre-post assignments that included interactive lectures through multimodal presentations and problem-based learning. Pre-post comparisons were done by paired t-test and Wilcoxon signed-rank test. Results: Pre-CME assessment showed low mean scores (4.04 [1.75]) in objective (assessment to be added) and limited perceived knowledge and ability. However, objective knowledge assessment showed a significant improvement (P

A Multimodal Course Design for Intercultural Business Communication

ABSTRACT This study presents a course design that integrates multimodal sounds, pictures, videos, student presentations, guest speakers, and group work into a multimodal course to enhance intercultural communication competence into a high-profile business training course. The course is tailor-made for 488 managers and staff in one of the world's top 10 banks whose branch is based in London. Innovative experiential techniques are embedded in the course to meet their professional needs. The evaluation from the participants suggested that the course was highly useful and challenging, which produced a positive impact on the trainees’ intercultural awareness and empathy in terms of more affective commitment, better behavioral performance and enhanced cognitive curiosity in intercultural business communication (IBC). The innovative ideas of the course design, examples of good practice and integrated multimodal pedagogy have paved the way for other colleagues designing and delivering similar courses.

Combining partial information from speech and text.

The current study investigated how partial speech and text information, distributed at various interruption rates, is combined to support sentence recognition in quiet. Speech and text stimuli were interrupted by silence and presented unimodally or combined in multimodal conditions. Across all conditions, performance was best at the highest interruption rates. Listeners were able to gain benefit from most multimodal presentations, even when the rate of interruption was mismatched between modalities. Supplementing partial speech with incomplete visual cues can improve sentence intelligibility and compensate for degraded speech in adverse listening conditions. However, individual variability in benefit depends on unimodal performance.

Starting young: Multimodal gender presentation in children’s pop music

A recent quantitative study (Smith, Choueiti, and Pieper 2018) demonstrates the hegemonic discrimination in today’s popular music scene, particularly but not exclusively in gender and race. This paper builds on that study, taking it not only into a multimodal dimension (where musical and visual performances are taken into account), but also extending it to children’s popular music, here defined as popular music performed by children for an audience and market primarily made up of children and their guardians. 
 The annual Norwegian popular music competition for children aged 8-15 Melodi Grand Prix Junior (MGP Jr) is the children’s equivalent of the adult competition to be Norway’s entry in the Eurovision Song Contest (ESC). It has been running since 2004, with 10 entries in each final, and those from 2007 onwards are available for public viewing on the national television channel’s website (tv.NRK.no). The resulting 130 songs thus provide a meaningful corpus from which to study current and recent multimodal gendered presentations of child performers. 
 Preliminary multimodal gender analyses (cf Maxwell and Mittner 2018) show that the performances are based around traditional gender binaries (i.e. boys and girls). While both presented genders sing, except for rare exceptions it is only boys who play instruments. This both complements and contrasts the study of the ESC (Isaksen forthcoming) which also shows a clear dominance of singing, particularly among female and female-presenting artists (including drag queens). 
 When these results from children’s pop music and from the ESC are set in relation to Smith, Choueiti, and Peiper 2018 in an interdisciplinary mixed methods approach, it is clear to see that the discrimination in the industry not only begins at a young age, it is also presented as normal, indeed attractive, to child viewers. This is borne out by the decreasing uptake of music tuition at Norwegian kulturskoler (the provider of state-sponsored lessons in the arts), particularly among school-age girls (Utdanningsdirektøret 2017). 
 In this paper I will present multimodal analyses of a selection of songs from MGP Jr in order to provide both examples of and exceptions to the norms shown by the statistics. In addition, an analysis of the (gendered) presentations of standard Norwegian instrument textbooks (cf Blix 2018) provides background context, with an emphasis on the gendered meanings that surround children in their everyday musical lives. 
 With thanks to Matilda Maxwell (age 11), aspiring instrumentalist, fan of MGP Jr, and research assistant.

Temporal dynamics of audiovisual affective processing

This study used event-related potentials (ERPs) to investigate the time course of auditory, visual, and audiovisual affective processing. Stimuli consisted of naturalistic silent videos, instrumental music clips, or combination of the two, with valence varied at three levels for each modality and arousal matched across valence conditions. Affective ratings of the unimodal and multimodal stimuli showed evidence of visual dominance, congruency, and negativity dominance effects. ERP results for unimodal presentations revealed valence effects in early components for both modalities, but only for the visual condition in a late positive potential. The ERP results for multimodal presentations showed effects for both visual valence and auditory valence in three components, early N200, P300 and LPP. A modeling analysis of the N200 component suggested its role in the visual dominance effect, which was further supported by a correlation between behavioral visual dominance scores and the early ERP components. Significant congruency comparisons were also found for N200 amplitudes, suggesting that congruency effects may occur early. Consistent differences between negative and positive valence were found for both visual and auditory modalities in the P300 at anterior electrode clusters, suggesting a potential source for the negativity dominance effect observed behaviorally. The separation between negative and positive valence also occurred at LPP for the visual modality. Significant auditory valence modulation was found for the LPP, implying an integration effect in which valence sensitivity of the LPP emerged for the audiovisual condition. These results provide a basis for mapping out the temporal dynamics of audiovisual affective processing.

Multimodal recognition of interrupted speech: Benefit from text and visual speech cues

Presenting degraded speech with visual cues facilitates speech recognition. This benefit is observed for visual speech cues that are perceptually correlated with the auditory signal, as well as for text cues that delay integration until a later cognitive-linguistic processing stage. However, it is not clear how the benefit compares between these two types of degraded multimodal presentations. The current study examined how listeners integrate visually interrupted text or visual speech cues with acoustically interrupted speech. In Experiment 1, text was periodically interrupted by white space at visual interruption rates that were associated with the auditory interruption rate of speech. In Experiment 2, videos were visually interrupted by grey frames. The synchrony of audio-visual interruption was also manipulated by presenting visual cues in-phase or 180o out-of-phase with speech interruptions. For both experiments, speech was low-pass filtered at 2000 Hz. Preliminary results indicate that listeners obtain a benefit from both visual speech and text cues. In addition, performance is affected by the interruption rate of speech, with minimal performance obtained around an interruption rate of 2 Hz. Supplementing speech with incomplete visual cues can improve sentence intelligibility and compensate for degraded speech in adverse listening conditions. [Work supported, in part, by NIH/NIDCD.]

A Responsive Pedagogical Initiative for Multimodal Oral Presentation Skills: An Action Research Study

Poor oral presentation skills amongst ESL speakers are frequently linked to weakness in general English proficiency. However, the literature has also identified the lack of a systematic and effective pedagogical method to develop students' multimodal oral presentation skills to meet 21 st century communication realities. This paper elucidates the application of a pedagogical initiative for multimodal oral presentations skills, informed by the Sociocultural Theory of Learning created by Vygotsky (1978), and the Pedagogy of Multiliteracies introduced by the New London Group (1996) and further developed by Cope and Kalantzis (2009). This initiative was experienced by 20 students through the action-reflection cycle methodology and was implemented for 13 weeks in a tertiary ESL classroom. Preliminary findings based on focus group interviews indicated that the initiative left a clear positive impact on student abilities in various multimodal components of oral presentation skills. Furthermore, the initiative positively affected students’ learning experience and raised their self-awareness of themselves as presenters. The findings also suggested that techniques employed in the initiative such as videos, feedback, collaborative learning and Web 2.0 could scaffold the students’ development of multimodal oral presentation skills. Based on these findings, significant implications for teaching oral presentation skills systematically are offered to be referred by ESL practitioners and researchers; in particular, the proposed multimodal oral presentations skills model as a multiliteracies pedagogy and the potential use of web 2.0 tools in promoting collaborative learning and creation of creative content by the learners. Keywords: ESL pedagogy; oral presentation skills; collaborative learning; multimodal literacy; action research

“Can There be a Full Moon at Daytime?” Young Students Making Sense of Illustrations of the Lunar Phases

ABSTRACTTeaching and learning situations nowadays typically build on richly illustrated material or multimodal presentations. Under these circumstances, the transparency of images and models used for explaining various phenomena becomes central. The present study deals with 20 Swedish children, 9–12 years old, discussing an illustration meant to show the cause of the different appearances of the Moon in the sky. The students’ task was to place eight numbered moon phases in the lunar orbit in the image. The illustration in question was chosen (a) because it was of a kind frequently used to explain the lunar phases and (b) because the phenomenon is known to be difficult to understand for students of all ages. The analysis leans on historical and sociocultural approaches as well as on multimodal semiotics. The results show that a majority of students were able to make sense of the most central features of the illustration but that very few spontaneously reasoned in a way that could be interpreted as the intended meaning‐making of the cause of the lunar phases. The results also indicate that the simultaneous adoption of two perspectives necessary for understanding the phenomenon was a stumbling block for most of the students.