Multimodal Experience Research Articles

SMEC: A card-based participatory design tool for seamless multimodal experience innovation in connected and autonomous vehicles

The diversity of multimodal interfaces and contextual complexity of designing multimodal interfaces call for novel participatory design approaches to facilitate co-creation between designers and stakeholders. This paper presents the SME design Cards (SMEC), a card-based participatory design tool developed for Seamless Multimodal Experience (SME) design in automobiles. We outline the tool’s background and development process, followed by presenting the evaluation results from both end-users and design experts. We then report on refinements to the SMEC and apply it in a hybrid workshop. We present insights into developing the card and discuss its effect in supporting SME design ideation and evaluation, informing the design of card-based participatory design tools.

Multimodal music datasets? Challenges and future goals in music processing

The term “multimodal music dataset” is often used to describe music-related datasets that represent music as a multimedia art form and multimodal experience. However, the term “multimodality” is often used differently in disciplines such as musicology, music psychology, and music technology. This paper proposes a definition of multimodality that works across different music disciplines. Many challenges are related to constructing, evaluating, and using multimodal music datasets. We provide a task-based categorization of multimodal datasets and suggest guidelines for their development. Diverse data pre-processing methods are illuminated, highlighting their contributions to transparent and reproducible music analysis. Additionally, evaluation metrics, methods, and benchmarks tailored for multimodal music processing tasks are scrutinized, empowering researchers to make informed decisions and facilitating cross-study comparisons.

Understanding Olfactory Perception and its Significance in The Exploration of Sensescape Scenarios in Asian Cities

Researchers and designers have begun to devote more time and energy to studying urban escape scenarios because they know how important it is to comprehend the complex ways in which people perceive and interact with the built environment. Olfaction, the sense of scent, is an important part of this multimodal experience since it greatly influences how perceive and engage with the environment. The importance of smell is magnified in Asian urban settings since these places frequently offer a varied and varied sensory landscape, marked by a multitude of environmental, cultural, and gastronomic influences. Architects, urban planners, and legislators may develop more inclusive and responsive natural settings that cater to the multimodal demands of various people by understanding the function of olfactory perception in constructing the sensescape in these cities.

Use of Technology in English Language Learning: Challenges and Benefits

The integration of technology in English language learning has gained significant attention due to its potential to enhance learning experiences and outcomes. However, its implementation also poses various challenges that need to be addressed. This study aims to explore the major challenges faced by teachers and students in integrating technology into English language learning classrooms, as well as the potential benefits it offers. A mixed-methods approach was employed, combining quantitative data from pre- and post-tests and qualitative data from classroom observations, interviews, and questionnaires. The findings reveal significant challenges, including limited access to technology, lack of digital literacy, and pedagogical considerations. Conversely, the benefits encompass increased motivation, exposure to authentic language resources, personalized learning, and multimodal experiences. By understanding these challenges and leveraging the advantages, educators and stakeholders can make informed decisions and implement effective strategies to optimize technology integration in language classrooms, ultimately enhancing learning experiences and outcomes.

Innovative Approaches in English Language Teaching: Integrating Communicative Methods and Technology for Enhanced Proficiency

This paper provides a comprehensive examination of effective English language teaching methodologies, focusing on communicative approaches and technology integration. Through an exploration of Task-Based Learning (TBL), Communicative Language Teaching (CLT), and Content-Based Instruction (CBI), as well as Computer-Assisted Language Learning (CALL), Virtual Reality (VR), and Mobile-Assisted Language Learning (MALL), this study evaluates their impact on language acquisition and proficiency development. The analysis highlights the role of authentic tasks, learner autonomy, and multimodal experiences in promoting meaningful communication and linguistic competence. Furthermore, it addresses the challenges and opportunities associated with integrating technology into English language instruction. By synthesizing current research findings and pedagogical practices, this paper offers insights into optimizing English language teaching in diverse educational contexts.

The impact of a single music therapy session on pain perception of patients with sickle cell disease

Sickle Cell Disease (SCD) is a genetic condition characterized by pain. The neuromatrix theory of pain suggests pain is a multimodal experience, recognizing the importance of interventions like music therapy (MT), which address the physical and emotional pain experience. This study examined the impact of a MT session on 21 hospitalized patients’ self-reported VOC and its overall benefits. Utilizing a convergent mixed methods design, pain scores were collected prior to, during, and after sessions, with a follow-up interview. Significant changes were observed. Patients reported an average 3.03-point difference between their pre-intervention and MT-intervention score (p<0.001). When pre- and post-intervention scores were compared, patients reported an average 1.9-point pain reduction (p<0.001). Five themes emerged: decreased pain, increased insight, openness in verbal engagement, improved mood, and expressive freedom. The ways in which patients engaged suggest the importance of simultaneously addressing the physical, cognitive and emotional pain experience. When patients became agents of their own pain management, it engaged them in a reflexive process that altered their pain perception, improved their mood, and changed their relationship with SCD. MT is beneficial in reducing pain, simultaneously addressing the multidimensional pain experience. Further research is needed to evaluate mechanisms of impact, including addressing medication dosage. Keywords: Music therapy, sickle cell disease, pain management, neuromatrix theory, hematology

PUPPET MAGIC: TRANSFORMING STORYTELLING INTO ENGLISH EXCELLENCE FOR ENGLISH YOUNG LEARNERS

The purpose of this project is to investigate how well puppetry works as a storytelling media for young learners, by focusing on what opinions do the students have about using puppets to present stories. knowing how puppetry functions in language learning can lead to more interesting and successful teaching strategies.This study used a qualitative way to answer the question.This research was carried out in SD Negeri 1 Muhammadiyah Banda Aceh.The writers believed that an interview was the best instrument to utilize because this study involved first-grade elementary school studentsUsing puppets as a medium takes advantage of first-graders’ receptiveness to multimodal experiences while acknowledging their developmental stage. The goal is to create a more dynamic and productive learning environment where the use of puppets stimulates students’ attention and helps them comprehend the stories that are being told.The overall perception of the young learners who involved in this study is that they enjoyed actively listening to the narrative story and closely paying attention when it incorporated media- puppet - into the story. This finding implies that it is crucial for English teachers working with young students to vary their concrete media, and using puppets is just one way to do this.

First-sight recognition of touched objects shows that chicks can solve Molyneux's problem.

If a congenitally blind person learns to distinguish between a cube and a sphere by touch, would they immediately recognize these objects by sight once their vision is restored? This question, posed by Molyneux in 1688, has puzzled philosophers and scientists since then. To overcome ethical and practical difficulties in the investigation of cross-modal recognition, we studied inexperienced poultry chicks, which can be reared in darkness until the moment of a visual test with no detrimental consequences. After hatching chicks in darkness, we exposed them to either tactile smooth or tactile bumpy stimuli for 24 h. Immediately after the tactile exposure, chicks were tested in a visual recognition task, during their first experience with light. At first sight, chicks that had been exposed in the tactile modality to smooth stimuli approached the visual smooth stimulus significantly more than those exposed to the tactile bumpy stimuli. These results show that visually inexperienced chicks can solve Molyneux's problem, indicating cross-modal recognition does not require previous multimodal experience. At least in this precocial species, supra-modal brain areas appear functional already at birth. This discovery paves the way for the investigation of predisposed cross-modal cognition that does not depend on visual experience.

How to build a more sustainable passenger air transport system: multimodal experience

A fully integrated passenger multimodal system is a new concept in the transport industry, and consequently, data on system interoperability is lacking. In this paper, a qualitative research method is applied, to get the transport experts’ opinions and information on how the multimodal system should be designed. Although they are aware that the collaboration should result in an increase in the volume of passengers, the most challenging task is to convince transport operators to share the data with each other, due to legal issues. To determine the passengers’ perspective on making multimodal choices in their journeys, the travel demand model is developed, based on online survey data. The proposed (Binary Logistic Regression) model reveals the importance for policymakers and transport operators serving the airport to address access time reliability by carefully considering the passengers’ preferences in terms of their needs and age.

Towards a Conceptual Model of Users’ Expectations of an Autonomous In-Vehicle Multimodal Experience

People are expected to have more opportunities to spend their free time inside the vehicle with advanced vehicle automation in the near future. This will enable people to turn their attention to desirable activities other than driving and to have varied in-vehicle interactions through multimodal ways of conveying and receiving information. Previous studies on in-vehicle multimodal interactions primarily have focused on making users evaluate the impacts of particular multimodal integrations on them, which do not fully provide an overall understanding of user expectations of the multimodal experience in autonomous vehicles. The research was thus designed to fill the research gap by posing the key question “What are the critical aspects that differentiate and characterise in-vehicle multimodal experiences?” To answer this question, five sessions of design fiction workshops were separately conducted with 17 people to understand the users’ expectations of the multimodal experience in autonomous vehicles. Twenty-two subthemes of users’ expected tasks of multimodal experience were extracted through thematic analysis. The research found that two dimensions, attention and duration, are critical aspects that impact in-vehicle multimodal interactions. With this knowledge, a conceptual model of the users’ in-vehicle multimodal experience was proposed with a two-dimensional spectrum, which populates four different layers: sustained, distinct, concurrent, and coherent. The proposed conceptual model could help designers understand and approach users’ expectations more clearly, allowing them to make more informed decisions from the initial stages of the design process.

Book review: Multimodal Experiences across Cultures, Spaces and Identities

Book review: <i>Multimodal Experiences across Cultures, Spaces and Identities</i>

INTEGRATION OF 3D SPATIAL INFORMATION FOR MULTI-MODAL EXPERIENCE OF THE URBAN ARCHIVE

Abstract. This paper focused on the case of Semal Village, the region that will be demolished due to undergoing recent redevelopment in Paju. We experimented utilizing 3D spatial information to capture precise urban morphology of present status before redevelopment and to transform such data into multi-modal experience of the urban archive. We investigated feasible techniques of photogrammetry and hybrid 3D modelling that could render the situated reality of the village communicatively with various medium such as 3D-printed scale model, and augmented reality (AR) contents. Through drone photography, the entire Semal village was captured, and three-dimensional data were obtained using photogrammetry. Along with information from on-site surveys, the mesh model was segmented into buildings, terrain, and vegetation for focused work. Each models reconstructed and retextured using images from the on-site surveys. All data were compiled for full-colour 3D printing and assembly. The 3D-printed scale model replicating Semal is on display at the Paju Central Library. Additionally, an AR content was created using the 3D-printed scale model and ethnographic data, aiming to archive and share people's memories, thereby continuously building a sustainable archive of the village.

Multimodal experience as a predictor and catalyst of travel behavior

Three surveys and an intervention experiment among university students investigated the relationship between past experience with multimodal transportation and current travel behavior. The case study university primarily draws students from rural, car-oriented communities but also attracts students from international and domestic urban cities where walking, biking, and public transit are common. Nearly 800 students responded to the first survey. They confirmed the hypothesis that rural students lack experience with multimodal travel and are less likely than their urban classmates to use multimodal opportunities in their new environment. Next, a subset of students was surveyed before and after participating in a multimodal exposure tour (the treatment group). Their responses were compared to students who completed the same surveys but did not participate in the tour. The results indicate that the treatment group increased their use of bus, bike, and walking. The exposure experiment should be interpreted with caution due to a small sample size, nevertheless, the results showed statistical and practical significance and were supported by the initial survey that involved a large, reliable sample. This case study is a microcosm of people moving to new mobility environments and the potential to transform travel habits. The survey design and tour event provide a model for investigating previous travel experience and promoting change in mode choice. The approach of this study can help planners, policymakers, business leaders, and university administrators better understand and shape the travel behavior of their constituents.

Perceptive vs reflective: spectator interpretation of multimodal artworks

This article presents an empirical investigation into spectator interpretation of multimodal artworks. Specifically, this article explores perception of the artwork in two different sensorial modalities — sight and sound — and the effect of their interaction. We selected four abstract paintings and created four acousmatic musical pieces that were composed to reflect the content of the artwork. We then ran a between-subjects experimental study in three conditions: visual only, music only, and a combination of both. A total of 48 participants completed an online survey in which they were asked to report their interpretations of the shown artworks. Following a thematic analysis on the collected data, we clustered participants’ interpretation into two main categories: reflective and perceptive. The combination increased spectators’ attention to the artworks, affected the temporality of the artworks, and created richer understandings of the multimodal works. The study provides knowledge for the inclusion of multimodal experiences in the presentation of art expanding the possibilities for inter-sensory dialogue in the arts.

Multilingual Learning Inside/Outside the Classroom: Insights from Intra‐Active Events

AbstractAs K‐5 dual language programs gain popularity in the United States, language teaching, however, often still prioritizes discrete and decontextualized learning that is not built on students’ interests and experiences or their multilingual and multimodal resources. This article reports on a classroom‐based ethnographic case study and illustrates how emergent bilinguals in a fifth grade dual language class engaged in a variety of multimodal experiences to understand local history through their first and second languages, community‐based inquiries, and writing. Drawing on translanguaging and new materialist understandings of language and literacy, the authors describe intra‐active events generated through museum activities and discussions in English and Chinese that allowed students to assemble whole‐body experiences and discussions in their journal writing about a day in the 18th century. The study shows how translanguaging can foster students’ engagement in learning as assemblages of their multilingual, multimodal, and multisensory experiences and support their writing processes and products, not only in a second language, but all language and literacy classrooms.

Kinesthetic Feedback for Understanding Program Execution.

To better prepare future generations, knowledge about computers and programming are one of the many skills that are part of almost all Science, Technology, Engineering, and Mathematic programs; however, teaching and learning programming is a complex task that is generally considered difficult by students and teachers alike. One approach to engage and inspire students from a variety of backgrounds is the use of educational robots. Unfortunately, previous research presents mixed results on the effectiveness of educational robots on student learning. One possibility for this lack of clarity may be because students have a wide variety of styles of learning. It is possible that the use of kinesthetic feedback, in addition to the normally used visual feedback, may improve learning with educational robots by providing a richer, multi-modal experience that may appeal to a larger number of students with different learning styles. It is also possible, however, that the addition of kinesthetic feedback, and how it may interfere with the visual feedback, may decrease a student's ability to interpret the program commands being executed by a robot, which is critical for program debugging. In this work, we investigated whether human participants were able to accurately determine a sequence of program commands performed by a robot when both kinesthetic and visual feedback were being used together. Command recall and end point location determination were compared to the typically used visual-only method, as well as a narrative description. Results from 10 sighted participants indicated that individuals were able to accurately determine a sequence of movement commands and their magnitude when using combined kinesthetic + visual feedback. Participants' recall accuracy of program commands was actually better with kinesthetic + visual feedback than just visual feedback. Although the recall accuracy was even better with the narrative description, this was primarily due to participants confusing an absolute rotation command with a relative rotation command with the kinesthetic + visual feedback. Participants' zone location accuracy of the end point after a command was executed was significantly better for both the kinesthetic + visual feedback and narrative methods compared to the visual-only method. Together, these results suggest that the use of both kinesthetic + visual feedback improves an individual's ability to interpret program commands, rather than decreases it.

Multisensory learning binds neurons into a cross-modal memory engram

Associating multiple sensory cues with objects and experience is a fundamental brain process that improves object recognition and memory performance. However, neural mechanisms that bind sensory features during learning and augment memory expression are unknown. Here we demonstrate multisensory appetitive and aversive memory in Drosophila. Combining colours and odours improved memory performance, even when each sensory modality was tested alone. Temporal control of neuronal function revealed visually selective mushroom body Kenyon cells (KCs) to be required for enhancement of both visual and olfactory memory after multisensory training. Voltage imaging in head-fixed flies showed that multisensory learning binds activity between streams of modality-specific KCs so that unimodal sensory input generates a multimodal neuronal response. Binding occurs between regions of the olfactory and visual KC axons, which receive valence-relevant dopaminergic reinforcement, and is propagated downstream. Dopamine locally releases GABAergic inhibition to permit specific microcircuits within KC-spanning serotonergic neurons to function as an excitatory bridge between the previously ‘modality-selective’ KC streams. Cross-modal binding thereby expands the KCs representing the memory engram for each modality into those representing the other. This broadening of the engram improves memory performance after multisensory learning and permits a single sensory feature to retrieve the memory of the multimodal experience.

Functions of (Illegal) Nightlife for Youth during the Pandemic: Between Neoliberalism and Neotribalism

The COVID-19 pandemic has had a great effect on young people, with their social lives being particularly impacted by the restrictions on social gatherings. Some of them, nevertheless, did not adhere to restrictions, and participated in illegal parties. In this study, we explored the functions of (illegal) nightlife for youth, as well as their motives for noncompliance with restrictions on social gatherings. We conducted in-depth semi-structured interviews with thirteen young people (seven male, aged 18-29) and the data was analyzed using reflexive thematic analysis. Restrictions on public gatherings revealed the multiple functions parties had for youth as a unique multimodal experience that can’t be easily substituted. Parties were described as unique experiences in particular because they facilitated the loss of individuality and coalescence with the collective. Some participants tried to make up for the loss of this resource by simulating parties in their own home, which proved to be unsatisfactory. Most participants refrained from going out, reporting a loss of a valuable resource for connecting with others and managing stress. In refraining from partying, some participants felt as though they lost a part of themselves. This highlighted the importance of the clubbing culture for defining one’s identity. Participants that continued to attend parties used them as a resource for managing stress related to the pandemic. Participants who attended illegal parties employed various strategies in order to relativize the seriousness of the pandemic. Nevertheless, most of them experienced a tension between their fear of the coronavirus and the need to feel carefree. Illegal partygoers also emphasized a greater sense of connectedness with other attendees than at pre-pandemic parties. Findings related to the social functions of parties for young people are interpreted from the perspective of theories of neoliberalism and neotribalism. We conclude that parties have primarily individualistic functions for young people, with distinct ‘patches’ of collectivism.

Sensed Presence, Attenuated Psychosis, and Transliminality: At the Threshold of Consciousness

Introduction: The experience of “sensed presence” or “felt presence” in the absence of “other” has been described as a complex multimodal experience to which meaning is given. Sensed presence (SenP) is a transdiagnostic experience that exists along a continuum that can appear during isolation, spirit quests, exposure to extreme elements, bereavement, anxiety, and psychosis. Given the prevalence and vast heterogeneity of SenP, in addition to a surprising lack of targeted research into this phenomenon, this research examined the interrelationship of SenP, attenuated psychosis symptoms (APS), and transliminality. Transliminality is composed of absorption, fantasy proneness, paranormal belief, mystical experiences, increased creativity, and hyperaesthesia. Methods: A completely anonymous online survey of unusual experiences and mental health was distributed via social media (i.e., Twitter, Facebook, Reddit, and mass emailing lists) to recruit participants. Demographic data were analyzed using χ² tests and one-way ANOVAs. A two-step cluster analysis was conducted to identify distinct sub-categories of transliminality followed by ANOVAs with bootstrapping at 1,000 iterations to compare SenP, increased APS, and transliminality. Pearson’s bivariate correlations were conducted to determine the association between SenP, APS, and transliminality. Results: Together with descriptive findings, we show distinct characteristics between clusters. T1 cluster consisted of individuals with few SenP experiences, low APS, and low transliminality. T2 consisted of individuals with a moderate prevalence of SenP, low APS, moderate transliminality, and increased overall feeling of closeness to G-d. There was no significant difference in APS between T1 and T2 or in the level of distress associated with APS. T3 individuals showed a significantly higher prevalence of SenP in all domains (frequency, distress, vividness, and total score), higher APS, and higher transliminality, compared to T1 and T2. The T3 cluster met criteria for high risk to develop psychosis. Conclusion: Thus, our findings demonstrate a strong association and entanglement of these experiences which suggests that the interrelatedness of transliminality/absorption and APS may serve as a potentially provocative underlying structure in the phenomenology of SenP.

The role of consciously timed movements in shaping and improving auditory timing.

Our subjective sense of time is intertwined with a plethora of perceptual, cognitive and motor functions, and likewise, the brain is equipped to expertly filter, weight and combine these signals for seamless interactions with a dynamic world. Until relatively recently, the literature on time perception has excluded the influence of simultaneous motor activity, yet it has been found that motor circuits in the brain are at the core of most timing functions. Several studies have now identified that concurrent movements exert robust effects on perceptual timing estimates, but critically have not assessed how humans consciously judge the duration of their own movements. This creates a gap in our understanding of the mechanisms driving movement-related effects on sensory timing. We sought to address this gap by administering a sensorimotor timing task in which we explicitly compared the timing of isolated auditory tones and arm movements, or both simultaneously. We contextualized our findings within a Bayesian cue combination framework, in which separate sources of temporal information are weighted by their reliability and integrated into a unitary time estimate that is more precise than either unisensory estimate. Our results revealed differences in accuracy between auditory, movement and combined trials, and (crucially) that combined trials were the most accurately timed. Under the Bayesian framework, we found that participants' combined estimates were more precise than isolated estimates, yet were sub-optimal when compared with the model's prediction, on average. These findings elucidate previously unknown qualities of conscious motor timing and propose computational mechanisms that can describe how movements combine with perceptual signals to create unified, multimodal experiences of time.