Multimodal Learning Environment Research Articles

Humor in multimodal language use: Students’ Response to a dialogic, social-networking online assignment

The focus of this study was on how students created and shared humor in an online multimodal learning environment. Across two semesters of a graduate course on psycholinguistics in education, 28 students posted 1006 pins on a social-networking webpage (Pinterest) that were subjected to a multimodal content analysis. The Pinterest boards provided a dialogic space for students to engage in on-going collaborative exploration of notable ways of using language in authentic sociocultural spaces. Many examples of humorous uses of language attested to students’ playful and creative application of course concepts. Through participants’ comments on pins and multimodal aspects of pins that transformed the nature of student-shared examples, some pins, originally not intended to be funny, came to represent examples of humor. Students took risks in pinning what they envisioned as possibly misconceived or even provocative content in pursuit of refining their understanding of psycholinguistic concepts.

STEM integrated education and multimodal educational material

The present article aims to underline the role of multimodal educational material in STEM Integrated early childhood education. Through social semiotics assumption that meaning arises in action and interaction, we argue that robotics, digital media, haptic materials, toys, books, tablets, actions, and artifacts have an active and dynamic role in multimodal learning and construct meaning in young children's STEM educational process. The literature review has revealed a research gap concerning combined multimodal aspects in STEM concepts for young children. We adopted a mixed-method collective case study design based on four case studies in which children interact with multimodal STEM educational material. Due to the principles for effective STEM teaching and the perspectives of integrated STEM education, our findings illustrate that MmEM in STEM concepts, through play-based, model-based, inquiry-based teaching practices (among other open-ended), may provide to children multimodal learning environments, engage them in authentic and meaningful learning, promote teamwork, communication and social skills, challenge and motivate them to make meaning of their learning.

Primary and lower secondary students’ perceptions of representational practices in science learning: focus on drawing and writing

ABSTRACT The study examined students’ perceptions related to emotional and cognitive engagement of linguistic and diagrammatic representational practices in science learning. Altogether, 251 Japanese sixth-grade (11–12 years old) and 187 ninth-grade students (14–15 years old) participated. The students' recognitions were measured by a self-report questionnaire and were scrutinised using the Rasch Rating Scale model. All participants valued both representational practices as important in science learning. Drawing was seen as a useful cognitive tool. However, participants did not have positive self-efficacy for either writing or drawing. While they lacked a confidence in their ability to write in science, had more positive beliefs of cognitive engagement in writing. Compared to the explanation activity, the students did not engage in representational practices in a metacognitive and communicative way, implying that they were not in a multimodal learning environment. The female students had less positive emotional engagement in science and representational practices than male students. The findings imply that diagrammatic conventions and procedural knowledge of using diverse representational forms should be emphasized in science curricula. Further research on the relationship between self-efficacy for representational practices, motivational engagements, and academic achievement is recommended, as well as cultural differences in student engagement in representational practices.

The virtual field trip: Investigating how to optimize immersive virtual learning in climate change education

AbstractImmersive Virtual Reality (IVR) is being used for educational virtual field trips (VFTs) involving scenarios that may be too difficult, dangerous or expensive to experience in real life. We implemented an immersive VFT within the investigation phase of an inquiry‐based learning (IBL) climate change intervention. Students investigated the consequences of climate change by virtually traveling to Greenland and exploring albedo and greenhouse effects first hand. A total of 102 seventh and eighth grade students were randomly assigned to one of two instructional conditions: (1) narrated pretraining followed by IVR exploration or (2) the same narrated training material integrated within the IVR exploration. Students in both conditions showed significant increases in declarative knowledge, self‐efficacy, interest, STEM intentions, outcome expectations and intentions to change behavior from the pre‐ to post‐assessment. However, there was a significant difference between conditions favoring the pretraining group on a transfer test consisting of an oral presentation to a fictitious UN panel. The findings suggest that educators can choose to present important prerequisite learning content before or during a VFT. However, adding pretraining may lead to better transfer test performance, presumably because it helps reduce cognitive load while learning in IVR. Practitioner NotesWhat is already known about this topic? Immersive virtual reality (IVR) simulations lead to higher presence but may lead to less learning when the content is not designed based on the affordances of the technology. One explanation for this finding is that cognitive load may be higher in IVR. The pretraining principle (ie, individuals learn more deeply from interactive multimodal learning environments when they receive pretraining on relevant prior knowledge) can be particularly effective in IVR‐based learning compared to learning through a video. Evidence shows that instructional design principles such as segmentation and generative learning strategies such as summarization can improve learning in IVR simulations. What this paper adds An investigation of the value of two different approaches to designing immersive virtual field trips (VFTs) within a real educational middle school context. Evidence that VFTs featuring IVR climate change simulations, in the context of inquiry‐based learning (IBL), can increase important variables such as declarative knowledge, interest in science and intentions to take climate action in seventh and eighth grade students. Evidence that presenting important learning content before a VFT leads to higher transfer scores. Implications for practice and/or policy Implementing an immersive VFT within the context of an IBL intervention provides students with relevant and engaging learning experiences and results in increased knowledge and interest in science. In the design of instruction using VFTs, educators can choose to either present prerequisite learning content prior to a VFT or during a VFT. However, adding pretraining has an advantage in terms of higher transfer scores.

‘It’s like being in bubbles’: affordances and challenges of virtual learning environments for collaborative learning in interpreter education

ABSTRACT We report on a study evaluating the educational opportunities that highly multimodal and interactive Virtual Learning Environments (VLE) provide for collaborative learning in the context of interpreter education. The study was prompted by previous research into the use of VLEs in interpreter education, which showed positive results but which focused on preparatory or ancillary activities and/or individual interpreting practice. The study reported here, which was part of a larger project on evaluating the use of VLEs in educating interpreters and their potential clients, explored the affordances of a videoconferencing platform and a 3D virtual world for collaborative learning in the context of dialogue interpreting. The participants were 13 student-interpreters, who conducted role-play simulations in both environments. Through a mix of methods such as non-participant observation, reflective group discussions, linguistic analysis of the recorded simulations, and a user experience survey several dimensions of using the VLEs were explored including the linguistic/discursive dimension (interpreting), the interactional dimension (communication management between the participants), the ergonomic dimension (human-computer interaction) and the psychological dimension (user experience, sense of presence). Both VLEs were found to be capable of supporting situated and autonomous learning in the interpreting context, although differences arose regarding the reported user experience.

The Reggio Emilia and the Mosaic approach: Opponents or allies in multimodal teaching and learning? A discussion of their contribution to multimodal learning in early years education

Multimodality is an important element of teaching and learning in early years settings. It provides opportunities for young children to communicate using different resources they feel comfortable with. This paper focuses on multimodality and multimodal learning in early years education and the different approaches, which can be used to create a multimodal learning environment. The purpose of this paper is to discuss how two powerful approaches, such as the Reggio Emilia and the Mosaic allow multimodal learning, which is crucial for children’s holistic development. The Reggio Emilia approach and the Mosaic approach are discussed, focusing on the strategies used to support children in their development using their multimodal ways of communication. The Reggio Emilia approach is analysed as an example that appreciates the artistic work of children (i.e., drawings to express their way of thinking) while on the other hand, the Mosaic approach is analysed as an example that supports children to have a leading role in their learning journey. This paper can be used as a starting point in raising awareness of practical multimodal teaching practices in early years education.

The impact of multimodal learning environments on the development of creativity

[Extract] In recent years there has been growing interest in Linguistic Landscape (henceforth LL) research that focuses on the use of languages in public spaces. The volume edited by Martin Putz and Neele Mundt is a valuable contribution to the expanding LL research that targets the visible representation of multiple languages in a globalized world. More specifically, the volume is a collection of 12 chapters which offer theoretical, methodological and empirical contributions to the field of LL, drawing on various disciplinary locations, such as linguistics, politics, sociology, literacy, education and cultural geography, and deploying quantitative as well as qualitative methods. The book has its origin in the 37th International LAUD Symposium held on 4-6 April 2016 at the University of Koblenz-Landau in Landau, Germany.

Integrating multimodal technologies with VARK strategies for learning and teaching EFL presentation: An investigation into learners’ achievements and perceptions of the learning process

Successful oral presentation effectually involves multi-faceted training of listening, writing, and nonverbal delivery besides speaking orally, which calls for a systematical holistic/multimodal approach. However, a multimodal learning environment for fostering EFL learners’ presentation development remains virtually unexplored. This study employed multimodal strategies adopted from the VARK model (visual, aural, reading/writing, and kinesthetic/gestural) with the support of digital audio, video, and speech visualization technologies in an English presentation course at a university in Taiwan. Two EFL classes served respectively as the experimental group with a technology-mediated multimodal approach and the control group with a traditional oral approach. Specifically, this research evaluated the experimental participants’ oral performance and explored their perceptions of this technology-mediated multimodal approach and its advantages and disadvantages as identified by the participants. Results from independent t-tests showed marginal significant progress of presentation performance in the experimental group. Descriptive statistics from the perception survey and content analysis of students’ reflective responses indicated that the participants were overwhelmingly positive about technology-supported multimodal activities implemented in the oral training course but encountered psychological and technological challenges when producing multimodal assignments. Theoretically, the results support the extension of multimodal theory to EFL oral presentation education. Practically the study informs EFL presentation instructors of the validity of technological-enhanced VARK strategies for learning and teaching EFL presentation. The research results also bear significant implications for the necessity of learner training on technology practices when pedagogy with the integration of multimodal technologies into EFL speech education is implemented.

A sequenced multimodal learning approach to support students' development of conceptual learning

AbstractVirtual learning environments can now be enriched not only with visual and auditory information, but also with tactile and kinesthetic feedback. However, the way to successfully integrate haptic feedback on a multimodal learning environment is still unclear. This study aims to provide guidelines on how visuohaptic simulations can be implemented effectively, thus the research question is: Under what conditions do visual and tactile information support students' development of conceptual learning of force‐related concepts? Participants comprised 170 undergraduate students of a Midwestern University enrolled in a physics for elementary education class. Four experiments were conducted using four different configurations of multimodal learning environments: Visual feedback only, haptic force feedback only, visual and haptic force feedback at the same time, and sequenced modality of haptic feedback first and visual feedback second. Our results suggest that haptic force feedback has the potential to enrich learning when compared with visual only environments. Also, haptic and visual modalities interact better when sequenced one after another rather than presented simultaneously. Finally, exposure to virtual learning environments enhanced by haptic forced feedback was a positive experience, but the ease of use and ease of interpretation was not so evident.

Creating a Multimodal Learning Environment to Enhance Grade 5 Learners’ Communicative Competence in English First Additional Language

Creating a Multimodal Learning Environment to Enhance Grade 5 Learners’ Communicative Competence in English First Additional Language

Enhancing multimodal learning through personalized gesture recognition

AbstractGestural recognition systems are important tools for leveraging movement‐based interactions in multimodal learning environments but personalizing these interactions has proven difficult. We offer an adaptable model that uses multimodal analytics, enabling students to define their physical interactions with computer‐assisted learning environments. We argue that these interactions are foundational to developing stronger connections between students' physical actions and digital representations within a multimodal space. Our model uses real time learning analytics for gesture recognition, training a hierarchical hidden‐Markov model with a “one‐shot” construct, learning from user‐defined gestures, and accessing 3 different modes of data: skeleton positions, kinematics features, and internal model parameters. Through an empirical comparison with a “pretrained” model, we show that our model can achieve a higher recognition accuracy in repeatability and recall tasks. This suggests that our approach is a promising way to create productive experiences with gesture‐based educational simulations, promoting personalized interfaces, and analytics of multimodal learning scenarios.

The integration of information in a digital, multi-modal learning environment

The aim of the reported eye-tracking study was to investigate whether learners integrate information presented on several pages within a digital learning environment and whether the underlying processes differ between a purely text-based and a multi-modal digital learning environment. Participants (N = 97) learned about the development of tornados. Picture presentation (yes vs. no) and presentation of inconsistent information between pages (yes vs. no) was varied between-subjects. Results showed that inconsistent information led to more intense gaze behavior. This indicates that learners re-activated information presented on previous pages to integrate it with the actual information, enabling them to detect the inconsistencies. Furthermore, the data pattern for text-related gaze behavior was not influenced by picture presentation, indicating that cognitive processes in purely text based or multi-modal learning environments are much the same. Comprehension outcomes did not differ between groups. The implications of these results are discussed.

Gesticulating science: Emergent bilingual students’ use of gestures

AbstractThis article examines how emergent bilingual students used gestures in science class, and the consequences of students’ gestures when their language repertoire limited their possibilities to express themselves. The study derived from observations in two science classes in Sweden. In the first class, 3rd grade students (9–10 years old) were involved in a unit concerning electricity. The second class consisted of 7th‐grade students (13–14 years old) working with acids and bases. Data were analyzed by using practical epistemological analysis (PEA). When students’ language proficiency limited their possibility to express themselves, using gestures resulted in the continuation of the science activities. Furthermore, both peers and teachers drew on the used gestures to talk about the science content. In some situations, the meaning of the gestures needed to be negotiated. Regardless, the gestures were always related to language. Both students and teachers participated in this process, but the teachers directed the communication toward the goal of the lessons: learning how to talk science. The study contributes to the field by showing the importance of paying attention to and valuing bilingual students’ use of gestures as a way to express scientific knowledge. In addition, it demonstrates how teachers might draw on students’ gestures to teach science and discusses the importance of creating multimodal learning environments. © 2017 Wiley Periodicals, Inc. J Res Sci Teach 55: 121–144, 2018

Engaging Students Through Multimodal Learning Environments: An Indonesian Context

<p>A critical issue regarding the English language teaching in the Asian context particularly in Indonesia is that the teaching and learning methods used are still far from being sufficient and effective to help students improve their language skill at the expected rate. This is a pilot study investigating the effects of multimodal learning environment on students’ learning outcomes connected to students’ writing performance. To find out the answers to the research problems, 60 students taking English for Specific Purposes at State Islamic University of Malang – Indonesia participated in this study. They were randomly selected and randomly assigned into the experimental group and the control group. The multimodal environment was provided to the experimental group whereas the control group was not. The results of the study indicated that the students in the experimental group had better writing performance. In addition, the contribution of this study is two folds; as a contribution to references on innovative and creative pedagogy in the classroom and to support the progress of the teaching and learning English.</p><p> </p><p><strong>Keywords: </strong><em>multimodal learning environments, video, the teaching of EFL writing, writing performance<strong></strong></em></p>

Problem-Based Learning in Multimodal Learning Environments

In this study, we enhanced a problem-based learning (PBL) environment with affordable, everyday technologies that can be found in most university classrooms (e.g., projectors, tablets, students’ own smartphones, traditional paper–pencil, and Facebook). The study was conducted over a 3-year period, with 60 postgraduate learners in a human–computer interaction course, following a PBL approach to teaching and learning. First, this article contributes a detailed description of how PBL can be enacted in a multimodal, technology-rich classroom. Second, the study presents evaluation data on learners’ technology adoption experience while engaging in PBL. Overall, the participants positively endorsed the learning environment, rating their experience highly on scales of communication and interaction, reflection, perceived learning, and satisfaction. In addition, quantitative content analysis of Facebook use documented how the physical and digital tools in the environment, coupled with the capability of Facebook as a recordkeeping and communication tool, were integral part of the PBL process.

Multimodal research: Addressing the complexity of multimodal environments and the challenges for CALL

AbstractMultimodality, the study of the interaction of language with other semiotic resources such as images and sound resources, has significant implications for computer assisted language learning (CALL) with regards to understanding the impact of digital environments on language teaching and learning. In this paper, we explore recent manifestations of CALL in 3-D virtual worlds, illustrated by the example ofSecond Life. The multimodal analyses of a conventional face-to-face lesson and three language learning activities inSecond Lifehighlight some of the affordances and challenges presented by 3-D virtual environments. The results suggest that while multimodal resources integrate naturally to facilitate language teaching and learning in an orderly, structured and goal-orientated manner in classroom lessons, the often uncoordinated use (or absence) of avatars’ gaze, facial expression, body posture, gesture, as well as the unclear proxemics and use of space pose problems for effective communication in a 3-D virtual world. In addition, a “technology-oriented” register, alongside traditional instructional and regulative genres and registers, is introduced to help students cope with the demands of learning a language in a 3-D virtual environment. The study raises the issue of the relative effectiveness of 3-D virtual worlds for language teaching and learning. In doing so, a digital approach to multimodal research is proposed in order to address the complexity of multimodal learning environments and the various challenges for CALL.

<b>Glossário hipermídia no ensino-aprendizagem implícito de vocabulário em língua inglesa

Considerando a carência de estudos sobre a influência da proficiência linguística no aprendizado implícito de vocabulário em ambiente hipermídia, este estudo investiga a relevância do glossário hipermídia no ensino-aprendizagem implícito de vocabulário de alunos de nível elementar de inglês como língua estrangeira. No plano teórico, o estudo é fundamentado nos pressupostos das seguintes teorias: Abordagem Conexionista, que busca explicar a cognição humana e a aquisição de conhecimento, e Teoria Cognitiva do Aprendizado Multimídia, proposta por Mayer (2001), que busca explicar o processamento da informação em ambiente de aprendizagem multimodal. Este estudo apresenta a seleção e o desenvolvimento de um ambiente hipermídia de leitura contendo um glossário hipermídia orientado por parâmetros delimitados no plano teórico. Esse material, utilizado na geração de dados, foi aplicado a 39 alunos de nível elementar de inglês da Universidade Federal de Juiz de Fora, divididos em dois grupos (A e B). O ganho médio de aprendizagem obtido tanto no Grupo A, 28,5 [p.p.], quanto no Grupo B, 21 [p.p.], indicam que o glossário hipermídia pode contribuir para o aprendizado implícito de vocabulário de alunos de nível elementar, com destaque para a anotação visual.

Building Bridges with the Parables: Developing Bilingual Students' EFL Skills and Cultural Awareness within a Multimodal Learning Environment

Building Bridges with the Parables: Developing Bilingual Students' EFL Skills and Cultural Awareness within a Multimodal Learning Environment

Berlin Kompass: Multimodal Gameful Empowerment for Foreign Language Learning

This article presents an innovative, gameful, multimodal, and authentic learning environment for training of oral communication in a foreign language—a virtual adventure called Berlin Kompass. After a brief presentation of the pedagogical and technological backgrounds, the system is described. Central results of a series of pilots in autumn 2013 with around 100 upper secondary pupils are described and further steps are discussed. The results are highly promising, and the concept was highly appreciated by the pupils regardless of some technical problems that the prototype had during the pilots. The researcher observations and the questionnaire results show that the concept manages to create a new motivating collaborative learning context with clear added value compared with equal tasks realized in a typical classroom approach.

Creative Multimodal Learning Environments and Blended Interaction for Problem-Based Activity in HCI Education

This exploratory case study aims to examine how students benefit from a multimodal learning environment while they engage in collaborative problem-based activity in a Human Computer Interaction (HCI) university course. For 12 weeks, 30 students, in groups of 5-7 each, participated in weekly face-to-face meetings and online interactions. Students’ selfreports and tutor’ observations showed that overall students benefited from the multimodal learning environment while they engaged in blended interactions across physical and digital tools and across collocated and networked learning spaces. With respect to tool use, it was found that the downward pointing projector and Facebook were used in all phases of problem based activity, while portable devices for record keeping were mainly used in progress evaluation and reflection phases.