Multimodal Design Research Articles

ПРОЯВЛЕНИЯ МУЛЬТИМОДАЛЬНОСТИ В ДИЗАЙНЕ

The article is devoted to variations of design multimodality within the framework of social semiotics and discourse theory, describing the simultaneity and holism of multimodal design texts in a socio-cultural context. The article gives a detailed analysis of multimodality in three directions: deep into the semiotic structure of the design product and the mental processes of its perception; in breadth – in numerous forms of organizing the interaction of communication participants; in time – in aspects of cultural citation. The author identifies and describes structural, citation, a priori multimodality and multimodal interaction. The a priori nature of multimodal perception and conventionality of multimodal design texts is considered as tools for achieving the goals of design communication.

Pictorial and Multimodal Metaphors in Informational Picture Books for Children <br/>—A Case Study

This article aims to explore how the use of pictorial and multimodal metaphors in informational picture books contributes to children’s understanding of the information or facts and, at the same time, attracts their interest in the plot of the story. The sample picture book selected for the analysis is Inside the Human Body, from The Magic School Bus series, the informational picture books written by Joanna Cole and illustrated by Bruce Degen. Based on Forceville’s categorization of pictorial and multimodal metaphors, the sample text is analyzed from three perspectives, namely, the construction of the narrative reality transmitted in this picture book, the identity construction of the main characters and the interaction of pictorial and multimodal metaphors in the successfully representation of the information to the young viewer in the book. The results of the analysis show that pictorial metaphors are employed in the construction of the narrative reality transmitted in the picture book to make the plot of the story interesting and reasonable; the picture book resorts to multimodal designs to construct the unique identities of its main characters both verbally and pictorially; the text writer and the illustrator of the information picture book collaborate to present the concepts and information through the combination of different types of pictorial and multimodal metaphors. The analysis also suggests that most of the pictorial metaphors are represented with two pictorially present terms (MP2) and verbo-pictorial metaphors (VPMs).

Preparing teachers to work with English learners: A multimodal vocabulary mini‐lesson project

AbstractIn K–12 classrooms, multimodal teaching, such as using visuals and technology, has been commonly embedded at different levels with different student groups. However, it has not been well discussed in English as a second language teacher training. As a response to the call for more research to study how teachers apply multimodal activities in working with English learners (ELs), this article reports a teacher educator’s effort to implement a multimodal vocabulary mini‐lesson project in one teacher education course that aims to train general education preservice teachers to work with ELs. The author describes the project and uses preservice teachers’ presentations, lesson plans, and project reflections as data to answer two questions: What are preservice teachers’ perceptions of designing multimodal teaching materials for all students, including ELs? How does the multimodal mini‐lesson design help preservice teacher candidates understand the key elements of educating ELs? Data indicate that teacher candidates showed positive attitudes toward designing multimodal lessons and materials, and the mini‐lesson appears to have deepened teacher candidates’ concepts of teaching ELs. A discussion of problems on multimodal design and potential solutions is also included.

IMARI: multi-Interventional program for prevention and early Management of Anastomotic leakage after low anterior resection in Rectal cancer patIents: rationale and study protocol

BackgroundAnastomotic leakage (AL) is still a common and feared complication after low anterior resection (LAR) for rectal cancer. The multifactorial pathophysiology of AL and lack of standardised treatment options requires a multi-modal approach to improve long-term anastomotic integrity. The objective of the IMARI-trial is to determine whether the one-year anastomotic integrity rate in patients undergoing LAR for rectal cancer can be improved using a multi-interventional program.MethodsIMARI is a multicentre prospective clinical effectiveness trial, whereby current local practice (control cohort) will be evaluated, and subsequently compared to results after implementation of the multi-interventional program (intervention cohort). Patients undergoing LAR for rectal cancer will be included. The multi-interventional program includes three preventive interventions (mechanical bowel preparation with oral antibiotics, tailored full splenic flexure mobilization and intraoperative fluorescence angiography using indocyanine green) combined with a standardised pathway for early detection and active management of AL. The primary outcome is anastomotic integrity, confirmed by CT-scan at one year postoperatively. Secondary outcomes include incidence of AL, protocol compliance and association with AL, temporary and permanent stoma rate, reintervention rate, quality of life and functional outcome. Microbiome analysis will be conducted to investigate the role of the rectal microbiome in AL.In a Dutch nationwide study, the AL rate was 20%, with anastomotic integrity of 90% after one year. Based on an expected reduction of AL due to the preventive approaches of 50%, and increase of anastomotic integrity by a standardised pathway for early detection and active management of AL, we hypothesised that the anastomotic integrity rate will increase from 90 to 97% at one year. An improvement of 7% in anastomotic integrity at one year was considered clinically relevant. A total number of 488 patients (244 per cohort) are needed to detect this difference, with 80% statistical power.DiscussionThe IMARI-trial is designed to evaluate whether a multi-interventional program can improve long-term anastomotic integrity after rectal cancer surgery. The uniqueness of IMARI lies in the multi-modal design that addresses the multifactorial pathophysiology for prevention, and a standardised pathway for early detection and active treatment of AL.Trial registrationTrialregister.nl (NL8261), January 2020.

Design of digital IIR filter: A research survey

This paper presents an overview on advancement made in designing of a digital infinite impulse response (IIR) filter. The design problem of an IIR filter was found to be challenging due to presence of poles in its transfer function. This makes the phase response of an IIR transfer function nonlinear and its magnitude response is also drifted due to quantization of coefficient values of denominator polynomials, which leads towards instability. Therefore, numerous efforts were made in order to acquire an optimal filter response using several optimization methods. Design problem of an IIR filter with various constraints was developed and solved using gradient based techniques, which resulted in optimal passband response with nearly linear phase or in some cases, absolute linearity was also achieved. However, the obtained solutions were sub-optimal in many cases due to transferring the multimodal design problem of an IIR filter into convex optimization. The solution was also affected due to the quantization of filter coefficients and in case of absolute linear phase response; a strong hick in magnitude response at beginning of transition edge frequency was obtained. To overcome the sub-optimality, researchers used evolutionary algorithms (EAs) for designing of an IIR filter. In time domain, system identification (SI) was adopted, whereas various error functions were developed in frequency domain for obtained magnitude responses close to desired response. This approach resulted in an optimal IIR filter response, however phase response linearity was not improved. Thus, EA approach was appropriated for lower order IIR filters. The design of various IIR filters like lowpass filter (LPF), highpass filter (HPF), bandpass filter (BPF) and bandstop filter (BSP) using an all-pass infinite impulse response (APF-IIR) was also reported. This approach is most appropriate, because the filter response is stable, nearly linear and magnitude response was also accurate. However, there is high error at the band edges of passband. Therefore, literature reveals that an APF based approach is most appropriate for various magnitude response filters.

Adaptation of a mobile phone health survey for risk factors for noncommunicable diseases in Colombia: a qualitative study

ABSTRACT Background Data collection on noncommunicable disease (NCD) behavioral risk factors has traditionally been carried out through face-to-face surveys. However, its high costs and logistical difficulties can lead to lack of timely statistics for planning, particularly in low and middle-income countries. Mobile phone surveys (MPS) have the potential to fill these gaps. Objective This study explores perceptions, feasibility and strategies to increase the acceptability and response rate of health surveys administered through MPS using interactive voice response in Colombia. Method A sequential multimodal exploratory design was used. We conducted key informant interviews (KII) with stakeholders from government and academia; focus group discussions (FGDs) and user-group tests (UGTs) with young adults and elderly people living in rural and urban settings (men and women). The KII and FGDs explored perceptions of using mobile phones for NCD surveys. In the UGTs, participants were administered an IVR survey, and they provided feedback on its usability and potential improvement. Results Between February and November 2017, we conducted 7 KII, 6 FGDs (n = 54) and 4 UGTs (n = 34). Most participants consider MPS is a novel way to explore risk factors in NCDs. They also recognize challenges for their implementation including security issues, technological literacy and telecommunications coverage, especially in rural areas. It was recommended to promote the survey using mass media before its deployment and stressing its objectives, responsible institution and data privacy safeguards. The preferences in the survey administration relate to factors such as skills in the use of mobile phones, age, availability of time and educational level. The participants recommend questionnaires shorter than 10 minutes. Conclusions The possibility of obtaining data through MPS at a population level represents an opportunity to improve the availability of risk-factor data. Steps towards increasing the acceptability and overcoming technological and methodological challenges need to be taken.

Doing expertise multilingually and multimodally in online English teaching videos

The use of online videos as a teaching resource is gaining importance. It opens up opportunities for the creation of knowledge, as educational content can now be accessed by anyone with an Internet connection. This democratisation of access to knowledge can also be seen in the language learning context, where English language teachers create online videos for a transnational audience. In this paper, we present a case study of how two online English teachers ‘do expertise’ in their lessons, drawing on their multilingual and multimodal repertoire so that expertise is talked into being. We conducted semi-structured interviews and analysed them by interpretative phenomenological analysis (IPA) to make sense of the teachers’ experiences of designing online teaching materials, and how their expertise was talked into being in the process. The aim of this paper is to contribute to understanding of expertise in the context of online language teaching. We argue that online teachers ‘do expertise’ by drawing on their multimodal design knowledge, pedagogical knowledge and linguistic knowledge.

A Multimodal Localization Framework Design for IoT Applications.

Multiple Radio Access Technology (multi-RAT) communication with Low Power Wide Area Networks (LPWAN) significantly increases the flexibility of Internet of Things (IoT) applications. Location-based services that build upon such a multimodal communication architecture are able to switch to an optimal localization method depending on the constraints of the active wireless technology. Furthermore, the resulting location estimate can aid location-based handover mechanisms to reduce the energy consumption of a multi-RAT IoT device. In this research, we present our design of a multimodal localization framework and illustrate the benefit of such a framework with two IoT use case examples. For the first use case, valuable artwork is tracked during transportation to a museum. In the second use case, we monitor the usage and location of large construction tools. Finally, we propose how our localization framework can be improved to deal with implementation challenges and to reduce location estimation errors.

Meta-atom library generation via an efficient multi-objective shape optimization method.

Optimizing the shape of metasurface unit cells can lead to tremendous performance gains in several critically important areas. This paper presents a method of generating and optimizing freeform shapes to improve efficiency and achieve multiple metasurface functionalities (e.g., different polarization responses). The designs are generated using a three-dimensional surface contour method, which can produce an extensive range of nearly arbitrary shapes using only a few variables. Unlike gradient-based topology optimization, the proposed method is compatible with existing global optimization techniques that have been shown to significantly outperform local optimization algorithms, especially in complex and multimodal design spaces.

Teaching Research on Ideology and Politics of "New Horizon College English" proceeding from multimodality

Based on the multimodal theory, this article designs a theme teaching in one unit of college English course ideology and politics which is about" Spend or save - The student's dilemma". The new instructional design incorporates President Mao ’s quotations, President Xi ’s speaking and habit of saving, and celebrity saving stories in regular college English teaching, and adds three specific multimodal designs of teachers’ teaching, students’ learning, and teaching evaluation. Multi-modal teaching mainly performs as various modal angles other than language and pictures, practical cases in teaching design.

Multimodal iron ore inbound logistics network design under demand uncertainty

ABSTRACT The paper considers the design of a multimodal and multilayer inbound logistics network for iron ore to be delivered from suppliers to steel plants. To make cost-efficient logistics and transportation planning decisions, steel companies should account for uncertainty in the demand for iron ore and the economies of scale during transshipment process in the port, which are two critical elements affecting decisions. To this end, we propose a two-stage nonlinear stochastic programming model with the first stage determining the choice of the port to perform transshipment operations and the needed capacity, and the second stage selecting transportation modes after demand uncertainty has been realized. To solve this problem, we first reformulate and linearize the model based on a quantity discount policy applied to the transshipment ports, and then, we develop a scenario-based decomposition algorithm. We conduct a case study based on the data of a steel company in China to illustrate the applicability of our proposed model. Moreover, we perform numerical experiments to demonstrate the effectiveness of our algorithm.

Designing Multi-Mode Power Split Hybrid Electric Vehicles Using the Hierarchical Topological Graph Theory

Power split hybrid electric vehicles (PS-HEVs) dominate the US and Japanese HEV market because of their superior fuel economy and drivability. In recent years, multi-mode PS-HEVs are offered by Toyota and GM. With multiple modes, it is possible for PS-HEVs to have both good launching/towing performance and fuel economy. Multiple modes are achieved by adding clutches or brakes. However, the corresponding design space can be quite large. To expedite the design process, a hierarchical topological graph theory approach is developed to systematically design a multi-mode PS-HEV with two planetary gear sets (PGSs). The process consists of three steps: 1) model the hybrid powertrain, 2) generate the multi-mode designs with specific modes, and 3) evaluate the performance of design candidates. In the performance screening process, designs are examined using the dynamic programming (DP) algorithm to evaluate their acceleration performance (0-100 km/h); and then a rapid dynamic programming (Rapid-DP) approach is used to compute their fuel economy under a specific driving cycle. Designs that pass the screening will then be retained as final vehicle designs. This design process ensures that the best designs are found and used.

Happy hearts do not hang down: the design process for the 2018 Valentine’s Day postage stamps of Finland

This article focuses on the multimodal design process for a set of Valentine’s Day postage stamps. It shows how semiotic resources were used to create the overall mood of the design. In particular, the style of the images made the general mood melancholic. During the process, the design was adapted to a commercial Valentine’s Day context through changes in specific resources.

Evolutions in the Multimodal Design of Italian Textbooks

Evolutions in the Multimodal Design of Italian Textbooks

Windows to the sold: verbo-visual multimodality in storefront windows

PurposeThe article seeks to enrich the body of research on store atmospherics by identifying how storefront window design impacts store entry decisions. An innovative multimodal design approach is presented, considering both visual and verbal constituents.Design/methodology/approachStudy 1 draws on a corpus of high-end storefront windows to create a categorization regarding different levels of verbo-visual complexity. The survey in Study 2 (n = 234) serves two purposes: first, to confirm these levels of complexity and second, to investigate the relation between the complexity of window design and store entry decisions.FindingsStudy 2 confirms the order of complexity established in Study 1. The results reveal an inverted-U relationship between window complexity and store entry propensity. Windows of medium level of complexity produce shoppers' relatively highest store entry propensity.Practical implicationsThe findings suggest that retailers would benefit from adopting verbo-visual window designs of medium complexity, as this combination optimizes the likeliness of consumers to enter stores.Originality/valueResearch on store atmospherics has until recently primarily focused on in-store cues. Studies on store windows remain vastly underrepresented in extant scholarship. The article not only fills this gap but also incorporates an original interdisciplinary angle on multimodality, which offers new methodological perspectives for research in retail and distribution scholarship.

Multimodal Continual Learning Using Online Dictionary Updating

This article studies the continual learning task from the perspective of multimodal fusion. In the multimodal fusion problem, the unified representations of heterogeneous modalities are continuously updated, while the classifier is also constantly refreshed for each multimodal learning task. To tackle this point, we establish a continual learning framework for multimodal learning and design an effective online dictionary updating method. Finally, we experimentally verify a complex material identification task and obtain promising results.

Luminescent Nanocomposites for Biomedical Applications

In this work, we pursue a concept of the multimodal drug preparation based on LuAG:Ce3+ nanoparticles. The functionality studied in this article is focused on the bioimaging of living cells via cathodoluminescence (CL) of scintillating nanoparticles attached to the cell surface. LuAG:Ce3+ scintillating nanoparticles were used as a core; amorphous or mesoporous SiO2 was used as a shell encapsulating the core to increase the surface reactivity; anti-Fas ligand (FasL) antibody was then attached to the outer shell in order to specifically bond the nanocomposite to the cell receptors; finally, polyethylene glycol molecules were attached to increase the biocompatibility of the nanocomposites. CL bioimaging was tested on yeasts Saccharomyces cerevisiae and leukocytes. It was found that the nanocomposites are able to attach to the cells via sorption or bonding to the receptors. The prepared nanocomposite shows stable luminescence under 10-keV electron irradiation, thus making a prospective candidate for this application and future multimodal drug design.

Aligning assessment, technology, and multi-literacies

This article is premised on research that suggests there is a gap between technology use for teaching and learning and the technology used in assessment. Digital technology such as blogs, Facebook, Twitter, podcasts, and news feeds are increasingly used in teaching and learning. On the other hand, assessment is often only in traditional essay form and frequently via pen and paper. This article examines the complex nature of multimodal design and argues for assessment that has constructive alignment, which sets out the interdependence of learning outcomes, teaching, and assessment where all three are aligned as equal partners. The aim of the research described was to investigate whether teacher–learners believed there was alignment between learning outcomes, activities, and assessment that utilized digital media technology. This article presents findings from a small-scale case study that drew on qualitative data from Student Evaluations of the Paper and from online surveys from four cohorts of participating teachers in a postgraduate adult literacy, language, and numeracy professional development program. There were three important findings from the data that indicated the importance of firstly having alignment of digital technology and course design outcomes, activities, and assessments and that this was clearly visible to the participants. The second finding indicated the content was relevant to the teacher–learners and involved them in constructing their own learning through authentic and practical activities and assessment. The third finding indicated that course outcomes, activities, and assessment tasks were aligned with multi-literacy skills. It is argued that academics design courses that are constructively aligned connecting learners with outcomes, activities, and assessment, which include an explicit focus on teaching the multi-literacy skills required in a world that is becoming increasingly digital.

Characterization, Simulation and Control of a Soft Helical Pneumatic Implantable Robot for Tissue Regeneration

Therapies for tissue repair and regeneration have remained sub-optimal, with limited approaches investigated to improve their effectiveness, dynamic and control response. We introduce a Soft Pneumatic Helically-Interlayered Actuator (SoPHIA) for tissue repair and regeneration of tubular tissues. The actuator features shape configurability in two and three dimensions for minimal or non-invasive in vivo implantation; multi-modal therapy to apply mechanical stimulation axially and radially, in accordance to the anatomy of tubular tissues; and anti-buckling structural strength. We present a model and characteristics of this soft actuator. SoPHIA reaches up to 36.3% of elongation with respect to its initial height and up to 7 N of force when pressurized at 38 kPa against anatomically-realistic spatial constraints. Furthermore, we introduce the capabilities of a physical in vivo simulator of biological tissue stiffness and growth, for the evaluation of the soft actuator in physiologically-relevant conditions. Lastly, we propose a model-based multi-stage control of the axial elongation of the actuator according to the tissue's physiological response. SoPHIA has the potential to reduce the invasiveness of surgical interventions and increase the effectiveness in growing tissue due to its mechanically compliant, configurable and multi-modal design.

Multimodal Language of Attitude in Digital Composition

Communication using popular digital media involves understanding multimodal systems of appraisal for expressing attitude, which traditionally deals with emotions, ethics, and aesthetics in language. The formulation and teaching of multimodal grammars for attitudinal meanings in popular texts and culture is currently underresearched. This article reports findings from multisite qualitative research that developed students’ ability to use semiotic resources for communicating attitude multimodally. The research participants were 68 students (ages 9–11 years) from two elementary schools. Students learned how to use attitudinal language—affect, judgment, and appreciation—and applied this knowledge to multimodal design. The findings advance a leading system of appraisal for discourse by adapting the system to the multimodal communication of attitude in digital comic making in schooling. The research is significant because it demonstrates the potentials for augmenting students’ linguistic and visual semiotic resources to convey multimodal attitudinal meanings in contemporary communication.