Multimodal Design Research Articles

Developing a Multimodal HMI Design Framework for Automotive Wellness in Autonomous Vehicles

With the development of autonomous technology, the research into multimodal human-machine interaction (HMI) for autonomous vehicles (AVs) has attracted extensive attention, especially in automotive wellness. To support the design of HMIs for automotive wellness in AVs, this paper proposes a multimodal design framework. First, three elements of the framework were envisioned based on the typical composition of an interactive system. Second, a five-step process for utilizing the proposed framework was suggested. Third, the framework was applied in a design education course for exemplification. Finally, the AttrakDiff questionnaire was used to evaluate these interactive prototypes with 20 participants who had an affinity for HMI design. The questionnaire responses showed that the overall impression was positive and this framework can help design students to effectively identify research gaps and expand design concepts in a systematic way. The proposed framework offers a design approach for the development of multimodal HMIs for autonomous wellness in AVs.

Sharpening Students' Racial Literacies through Multimodal Subversion

AbstractThis case study shares the experiences of 24 students in an urban high school in the U.S. Midwest who spent 6 weeks learning about, and applying tenets of, critical race theory (CRT) to their analysis of the canonical The Adventures of Huckleberry Finn. Students created a visual essay reflecting their understanding of CRT in the novel and in their broader social milieu. Drawing from, and sharing possibilities for, merging theoretical principles from CRT, multiliteracies, and the writing process, we/the authors ask: How can students utilize multiple perspectives to sharpen racial literacies through multimodal design? These findings include three points of integration: (a) overt instruction, Whiteness, and organizing; (b) critical framing, intersectionality, and address; and (c) transformed practice, critical stance, and revise and survey again.

Ejector mode performance improvement of rocket-based combined cycle engine designed for dual-ramjet mode

Rocket-based combined cycle engine integrating multi-mode operation accomplishes an excellent performance under various flight speed conditions. Generally researchers selected dual-ramjet mode as the cruise and design point with adjusting heat addition method to improve the performance under other modes. To analyze the difficulties during the multi-mode design, a unilateral expansive flowpath through central rocket ejection was investigated to analyze the effects of the mixing ratio of oxygen and fuel and the mass flow rate of the primary rocket plume. The results indicate that the mixing ratio increase can significantly strengthen the entrainment amount. There is an optimal operating rate to obtain the highest entrainment ratio due to Fabri choking limit induced by a confined path width. However, the excessively divergent afterbody based on dual-ramjet results in the under-expansion issue of the exhaust gas and negative thrust of the nozzle. The multistage ejection method can effectively overcome this shortage by compressing the exhaust gas. But with the flight speed increasing, the existence of upstream bow shock induces high inlet resistance. Adding extra duct can effectively solve the problem by decreasing inlet convergent ratio and sucking more secondary flow for combustion.

No laughing matter!? Analyzing the Page Layout of Instruction Comics

ABSTRACT The study reported in this paper analyzes the specific genre of ‘instruction comics’ with regard to typical patterns of comic book pages. As a sequential-pictorial genre with a narrative as well as instructive character, ‘instruction comics’ have recently and increasingly been used in education as a novel way to communicate, for instance, health-related information. The corpus for this study includes 37 single-page first-aid comics that all show how to help a person who is choking, i.e. how to perform what is generally known as the Heimlich manoeuvre. To understand how this procedure is presented visually and how this presentation is similar to the layout of typical pages of comic books and graphic novels, the corpus is annotated with a multi-level annotation scheme that has been developed to annotate and analyse the page layout of visual narratives. The annotation scheme includes formal descriptions to synergistically analyse the complexity and variety of various comic genres, with specifications for typical comic page grids and the use of framing, inset panels, or captions. The paper presents and critically discusses the results of the corpus annotation with regard to the specific multimodal design choices in the corpus of choking comics.

Rhythm in literary apps

This article addresses how rhythm may function in literary apps. The article has two aims: increasing the knowledge of how literary apps work as texts, by exploring their aspects of rhythm, and developing the understanding of the theoretical term of rhythm. The authors propose a rhythmanalysis in which two different types of rhythm – reading rhythm and narrative rhythm – are taken into account. The two types of rhythm may both occur at different structural levels in the text. This approach is applied to the analysis of rhythm in the popular literary app, Florence (Wong et al., 2018, Florence Tablet application software), drawing on concepts from multimodal social semiotics (Van Leeuwen, Introducing Social Semiotics, 2005), although leaning towards a more reception-oriented approach than the traditional text-oriented analysis in social semiotics. Literary apps are defined in this context as multimodal fictional narratives that can lead to an aesthetic experience for the reader (Iser, 1984, Der Akt des Lesens); however, non-narrative apps, such as poetry, may also be defined as literary apps. These apps may be read on a tablet or a smartphone. This article elucidates some of the many facets of rhythm related to the multimodal design of a literary app, which invites different forms of interactivity than the linear reading and page-turning of print-based picture books. The findings of the analysis show how rhythm not only contributes to the multimodal cohesional aspects of literary apps, but is fundamental to the meaning potential of the literary app.

Emerging Practices in Multimodal Design and Performance Measurement: Review of Recent Literature and Practical Documents

This paper summarizes the state of the knowledge and practice for multimodal design and the use of multimodal performance measures in transportation projects. The traditional auto-centric approach to transportation performance measurement is reviewed before turning to recent efforts to design and evaluate projects across multiple modes. Approaches to developing multimodal level of service (MMLOS) measures, including the differences between supply- and demand-based performance measures, are reviewed and how these measures can integrate into the design process summarized. Five U.S. state Departments of Transportation are identified as multimodal practice leaders and a review of their design processes is provided to benchmark the state of the practice. It is found that, compared with traditional approaches, modern best practice for multimodal projects engages a broader range of stakeholders in project design, considers a wider range of alternatives, and allows for more design flexibility to match a solution to its specific context. The literature and current practice have not yet standardized on a single approach or list of performance measures to fit all multimodal projects. Rather, today’s multimodal design processes employ a suite of performance measures that address the competing stakeholder interests involved in a solution and allow them to see the tradeoffs among competing goals.

MULTIMODAL PEDAGOGICAL DESIGN AND LESSONS IN THE PHILOSOPHY OF EXISTENTIALISM (ON THE EXAMPLE OF A. ZVYAGINTSEV'S FILMS)

The authors of the article consider the concept of multimodal pedagogical design in the works of the founders of this concept M. Kalantzis and B. Cope. They connect the necessity of multimodal pedagogical design implementation with the formation of new, non-didactic digital pedagogy and the need to strengthen the emotional involvement of students, especially those from another culture and need to be immersed in a socio-cultural context. The experience of multimodal pedagogical design application is analyzed on the example of using films and website of Russian director A. Zvyagintsev, who develops traditions of existential cinematography and creates complex intertextual and culture-centered artistic language in his film narratives. The authors involve scientists from the Center for Media Education (Taganrog), Ural Federal University, and foreign colleagues to discuss the potential of cinematography as one of the instruments of multimodal pedagogical design around which the study of certain disciplines can be built. The conclusion is that only by engaging modern multimodal means of transferring knowledge and transforming existential experience, such as modern cinema and social networks, websites, blogs, it is possible to successfully attract and retain the attention of the audience in the remote education process.

Speculative Design as a Collaborative Practice: Ameliorating the Consequences of Illiteracy through Digital Touch

This article and the design fictions it presents are bound up with an ongoing qualitative-ethnographic study with Imazighen, the native people in remote Morocco. This group of people is marked by textual and digital illiteracy. We are in the process of developing multi-modal design fictions that can be used in workshops as a starting point for the co-development of further design fictions that envision the local population's desired digital futures. The design fictions take the form of storyboards, allowing for a non-textual engagement. The current content seeks to explore challenges, potentials, margins, and limitations for the future design of haptic and touch-sensitive technology as a means for interpersonal communication and information procurement. Design fictions provide a way of exposing the locals to possible digital futures so that they can actively engage with them and explore the bounds and confines of their literacy and the extent to which it matters.

Evaluating multimodal literacy: Academic and professional interactions around student-produced instructional video tutorials

This paper offers a reflection on how academic and professional interactions can help guide best practices for constructing viable evaluation grids to assess multimodal literacy. Preparing English-language learners for today's digitally and culturally complex workplace environment is a central concern in English as a second language (L2) teaching environments. It requires meeting specific teaching goals, such as supporting traditional print and multimodal literacies as well as increasing learners' English-language fluency and appropriateness. Our study focuses on an underexplored professional multimodal genre – instructional video tutorials – and proposes a multimodal evaluation grid incorporating theoretical concepts and empirical results from multimodal linguistics and multimedia learning. We examine how four video communication professionals use the grid to measure the effectiveness of students' video tutorials and identify areas for improvement. We present results for three areas for which the experts considered students did not meet expected professional standards: information organization, timing, and L2 spoken language narration. Our findings suggest possibilities for introducing appropriate forms of action or intervention into teaching multimodal design projects to better prepare L2 English students to meet workplace multimodal literacy requirements.

Fonts That Fit the Music: A Multimodal Design Trend Analysis of Lyric Videos

Fonts That Fit the Music: A Multimodal Design Trend Analysis of Lyric Videos

Autism Multimodal Activism. Negotiating a Pallet of Identities on the YouTube Social Media

Background: Despite recent developments in the field of multimodality and disability there are only a few attempts to incorporate its theoretical developments for the study of disability and autism. Purpose: The present paper addresses this gap by attempting to explore the multimodal practices of an autism activist in a video posted in the popular YouTube media. Method: Drawing on an integrative multimodal design, it proposes a novel exemplar for identifying the complex interplay between the identities pronounced and their association with the underlying multimodal resources used. Results: Through a detailed analysis, the paper shows how the activist enacted a range of identities situating herself in differential positions with regards to the mainstream understanding of autism. Examining the role of the wider semiotic resources used but also the interplay between them, the paper tries to reshape the understanding of the social actions depicted by the activist. The use of such means as a way to contest the conventional societal views about autistic activities is discussed. Conclusions: Overall, the paper considers the intricate relationship that exists among meaning making resources, their use in multimodal (inter)action and the inevitability of escaping hegemonic discourses which secondarily delimit autistic lives through their pathological associations.

Size exclusion chromatography coupled small angle X-ray scattering with tandem multiangle light scattering at the SIBYLS beamline.

Small-angle X-ray Scattering (SAXS) has been a versatile technique for studying biomolecules in solution for several decades now. Developments in SAXS techniques that integrate in situ purification with a high-throughput, multimodal design philosophy have revolutionized the reach and tempo of BioSAXS experiments. The current zenith of the field comes in the form of size exclusion chromatography coupled SAXS with in-line multiangle light scattering (SEC-SAXS-MALS). This technique has been a considerable focus at the Structurally Integrated BiologY for Life Sciences (SIBYLS) beamline at the Advanced Light Source (ALS) in Berkeley, California, over the last 5 years and continues to be a point of active development. In this chapter, we describe the design of the SEC-SAXS-MALS system and general guidelines for collecting, processing, and analyzing SEC-SAXS-MALS data at the SIBYLS beamline.

Turn-taking in collaborative storytelling

This conversation-analytic paper investigates the multimodal design and interactional functions of the connective et puis après (‘and then after that’) in a French-language corpus of video-recorded collaborative storytellings. Two similar, yet different, sequential positions are investigated: the juncture between subsequent story episodes and the space between extended side sequences and the return to the story-in-progress. Such juncture positions constitute recognizable moments at which both members of the telling party, i. e., the current teller and the co-teller, must determine the topic of the next story episode as well as its delivery. Thus, juncture positions provide a perspicuous setting for the analysis of how tellership is negotiated and how topic progression is achieved. The connective et puis après appears to be a resource for current tellers to establish spaces for coparticipation at juncture positions, closing prior talk and projecting continuation. The multimodal analysis shows that both its prosodic design and co-occurring changes of the embodied participation framework contribute to opening interactive turn spaces and to making telling-specific next actions relevant.

Detecting Emotions through Electrodermal Activity in Learning Contexts: A Systematic Review.

There is a strong increase in the use of devices that measure physiological arousal through electrodermal activity (EDA). Although there is a long tradition of studying emotions during learning, researchers have only recently started to use EDA to measure emotions in the context of education and learning. This systematic review aimed to provide insight into how EDA is currently used in these settings. The review aimed to investigate the methodological aspects of EDA measures in educational research and synthesize existing empirical evidence on the relation of physiological arousal, as measured by EDA, with learning outcomes and learning processes. The methodological results pointed to considerable variation in the usage of EDA in educational research and indicated that few implicit standards exist. Results regarding learning revealed inconsistent associations between physiological arousal and learning outcomes, which seem mainly due to underlying methodological differences. Furthermore, EDA frequently fluctuated during different stages of the learning process. Compared to this unimodal approach, multimodal designs provide the potential to better understand these fluctuations at critical moments. Overall, this review signals a clear need for explicit guidelines and standards for EDA processing in educational research in order to build a more profound understanding of the role of physiological arousal during learning.

An N-Port Universal Multimode Optical Router Supporting Mode-Division Multiplexing.

Optical network-on-chip (ONoC) is based on optical interconnects and optical routers (ORs), which have obvious advantages in bandwidth and power consumption. Transmission capacity is a significant performance in ONoC architecture, which has to be fully considered during the design process. Relying on mode-division multiplexing (MDM) technology, the system capacity of optical interconnection is greatly improved compared to the traditional multiplexing technology. With the explosion in MDM technology, the optical router supporting MDM came into being. In this paper, we design a multimode optical router (MDM-OR) model and analyze its indicators. Above all, we propose a novel multimode switching element and design an N-port universal multimode optical router (MDM-OR) model. Secondly, we analyze the insertion loss model of different optical devices and the crosstalk noise model of N-port MDM-OR. On this basis, a multimode router structure of a single-mode five-port optical router is proposed. At the same time, we analyze the transmission loss, crosstalk noise, signal-to-noise radio (OSNR), and bit error rate (BER) of different input–output pairs by inputting the 1550 nm TE, TE, and TE modes to the router.

Process Product Design Based on Multimode Interaction Design

There are many problems in the current process product design, such as poor interaction and low overall efficiency, so a process product design method based on multimode interaction design is proposed. Based on the relationship model, environment model, and object-oriented model of process product design, the process of process product development and design is analyzed. On this basis, the finite state machine is used to analyze the state of the designer to complete the task sequence, as well as the related design task planning, constructing craft products multimodal interaction mechanism, with the improved fuzzy analytic hierarchy process to evaluate the quality of the products, according to the evaluation results of technology of product design optimization. The experimental results show that both experts and ordinary users give a high evaluation of the product designed in this paper, and the design cycle of the product is significantly shortened, indicating that this method can fully save the design time and improve the overall efficiency.

Effect of mapping characteristic on audiovisual warning: Evidence from a simulated driving study

Advanced driver assistance systems (ADAS) can enhance road safety by sending warning signals to drivers. Multimodal signals are gaining attention in ADAS warning design because they offer redundant information that facilitates human–system communication. However, no consensus has been reached on which multimodal design offers optimal benefit to road safety. Icons iconically map the real world and are associated with fast recognition and response time. Therefore, this study aims to investigate whether visual and auditory icons will benefit the effectiveness of audiovisual multimodal warnings. Thirty-two participants (16 females) experienced four types of unimodal warnings (high and low mapping visual warnings and high and low mapping auditory warnings) and four types of audiovisual warnings (high mapping visual + high mapping auditory warning, low mapping visual + low mapping auditory warning, high mapping visual + low mapping auditory warning, and low mapping visual + high mapping auditory warning) in simulated driving conditions. Visual warnings are presented in a head-up display. Results showed that multimodal warnings outperformed unimodal warnings (i.e., modality effect). We found mapping effect in audiovisual warnings, but only high mapping auditory constituents benefited warning effectiveness. Eye movement results revealed that the high mapping constituents might distract drivers from the road. This study adds evidence that multimodal warnings can offer extra benefits to drivers and high mapping auditory signals should be included in multimodal warning design to achieve better driving performance.

Towards a unified affordance approach: searching for congruent meaning making in COVID-19 warning designs

ABSTRACT This case study on COVID-19 warning designs in a Swedish context illustrates how a unified affordance approach may contribute to an understanding of the meaning-making in reminders, instructions, cues and prompts that communicate the message “keep your distance.” The analysis combines semiotic and ecological affordance categories, taking both Gibson’s original theorizing on affordances and more recent affordance-informed research efforts into consideration. In so doing, the study aims to bridge a knowledge gap in the study of visual instructions and warning designs as well as in a more comprehensive way delineate the multimodal design strategies associated with COVID-19 warning designs. The analysis shows that Swedish COVID-19 warning designs of the keep-your-distance-kind belong to a non-standardized and emerging genre that is marked by great variation and ad-hoc design solutions, several of which combine physical blocking functions with verbally based messages. The analysis also highlights the tension between verbal and visual recourses, on one hand, and the signage placement and choice of materials, on the other hand. It is concluded that communication resources do not always appear to convey the same basic message, but in incongruent ways weaken what might be considered the intended main message.

Wheelchair Control With Inductive Intra-Oral Tongue Interface for Individuals With Tetraplegia

High level spinal cord injury drastically reduces the quality of life of the injured person. Various systems attempted to interface the still intact or residual abilities following injury in various monomodal or multimodal designs to compensate for the highly affected mobility. An intra-oral inductive tongue computer interface (ITCI) has been designed to provide real-time discrete and proportional control for computers and assistive devices and to meet specific requirements for individuals with tetraplegia. Operation of the ITCI for wheelchair control was demonstrated with two participants with tetraplegia in a short-term training study. Additionally, two non-disabled individuals participated in the study. For each participant, the ability to drive a Permobil C500 with the ITCI was compared to that when driving the wheelchair with joystick (mouth-stick in one case) along two different lanes of 39 m, by reporting the speed along the lanes and the number of obstacles hit. The lanes consisted of 90°, 360°, and complex maneuver segments linked by linear segments. The ITCI featured a mouthpiece encapsulating two pads of inductive sensors, driving electronics, and battery. The mouthpiece was attached to the palate of the participant’s oral cavity with dental retainers. A piercing-like activation unit was attached to the tongue. Data were transmitted wirelessly to a central unit that controlled the wheelchair through wired interface. Among all participants, mean speeds along lane A or B reached maximal values between 0.42 and 0.74 m/s when driving with the ITCI, representing 41 to 71% of that obtained when driving with the joystick.

Cut to the chase – How multimodal cohesion secures narrative orientation in film trailers

Recent years have seen a growing number of studies explore the narrative, persuasive and multimodal design of film trailers (Maier, 2009, 2011; Wildfeuer and Pollaroli, 2017). While such work has primarily looked at the ways in which trailers realize persuasive functions (Hediger, 2001; Maier, 2011; Krebs, 2020), this paper examines how the (audio-visual) representational choices made by trailer editors can provide narrative orientation for the film audience. Inside a self-compiled corpus of 150 genre-stratified US-American film trailers, recurring audio-visual, cohesive patterns are elicited. It will be shown that the visual effects of this cohesive thrust can be tied to the cinematographic choices editors make in the opening shots of the trailer, by using specific shot scales, (types of) settings and (non-)human represented participants. To capture the verbal dynamics of trailer cohesion, the study explores the key semantic domains to which all lexical expressions in the trailer dialogues can be linked (Rayson et al., 2004). Results of this study confirm that both visual and verbal elements form recurring genre-specific patterns. They unleash their multimodal potential of film trailers by tapping into the previous (genre) knowledge of trailer audiences, facilitating the cognitive uptake of audiovisual information for the audience as the trailer unfolds.