Multimodal Learning Strategies Research Articles

Predicting the number of COVID-19 imported cases based on cross-modal transformer: A case study in China

With the global outbreak of COVID-19, an increasing number of countries have made imported epidemic control a priority, imposing restriction measures to prevent the spread of the virus caused by imported cases. To control the imported epidemic, it is necessary to accurately predict the number of imported cases from different source countries. This paper proposes a novel time series prediction approach called PNICA (Prediction on Number of Imported CAses) that uses deep learning to predict the number of COVID-19 imported cases. On the one hand, the proposed PNICA approach adopts a multi-modal learning strategy to fuse three sources of data: flight data, the epidemic data, and the data of historical imported cases. On the other hand, the proposed PNICA approach extends the traditional transformer model with cross-modal attention to learn the interactions between different data modalities to improve prediction accuracy. We use China as the target country and collect the number of imported cases from four source countries—Japan, USA, Russia, and the UK—as well as the epidemic data and flight data from May to November 2020. Experiments on the collected data demonstrate that the proposed PNICA approach outperforms the baseline methods in predicting the number of imported cases. The ablation study shows that both the multi-modal learning strategy and cross-modal attention can significantly improve prediction performance.

Multimodal Learning Strategies in Secondary EFL Education: Insights from Teachers

Multimodal learning has shown promise in various educational settings, positively impacting teaching and learning outcomes. However, the multifaceted effects of multimodal learning on secondary EFL students' second language acquisition still need to be explored, especially at the secondary school level. A deeper understanding of how multimodal learning—which involves the integration of linguistic, auditory, visual, and kinesthetic modalities to convey information—influences second language learning is crucial for effective language teaching. The study aims to explore the role and instructional practices of multimodal learning in enhancing students' second language outcomes through in-depth interviews with teachers. Eight English teachers from several junior high schools across East Nusa Tenggara, Indonesia, participated in the study. The participants had varied teaching experiences: three teachers with 6–10 years, four with 11–15 years, and one with over 16 years of experience. The results indicate that teachers have diverse experiences integrating various combinations of learning modes into their teaching practices. Four key themes emerged regarding the role of multimodal learning in students' second language acquisition, enhancing understanding, improving retention, encouraging participation and motivation, and accommodating different learning styles. This study offers insights for teachers to optimize their teaching strategies and highlights the effectiveness of multimodal approaches in English language education.

Design and Application of English-Chinese Bilingual Teaching Model Based on Multimodal Learning

In the globalized landscape of education, the demand for effective bilingual teaching models has become increasingly prominent, particularly in English and Chinese instruction. this study introduces the Bilingual Teaching Random Conditional Field (BTRCF) as a novel approach to designing and implementing an English-Chinese Bilingual Teaching Model based on Multimodal Learning. The principles of multimodal learning, which integrates various modes of communication such as visual, auditory, and kinaesthetic, the BTRCF model aims to provide students with a comprehensive and immersive language learning experience. The BTRCF model capitalizes on the complementary strengths of both English and Chinese languages, aiming to cultivate bilingual proficiency while fostering cross-cultural understanding among students. By incorporating elements from each language and leveraging multimodal learning strategies, the BTRCF model provides students with a comprehensive language learning experience that extends beyond mere linguistic proficiency to encompass cultural appreciation and intercultural competence. By capitalizing on the complementary strengths of both languages, this model seeks to foster bilingual proficiency while promoting cross-cultural understanding among students. with the BTRCF model have shown promising results, with an average increase of 25% in students' bilingual proficiency scores compared to traditional teaching methods. Additionally, qualitative assessments reveal high levels of student engagement and satisfaction with the multimodal learning approach employed by the BTRCF model, indicating its potential to revolutionize bilingual education practices.

Efficacy of Collaborative and Multimodal Learning Strategies in Enhancing English Language Proficiency Among ESL/EFL Learners: A Quantitative Analysis

This study examined the efficacy of collaborative and multimodal learning strategies in enhancing English language proficiency among ESL/EFL learners. A total of 120 secondary school-aged learners from three schools in the United Kingdom participated in the study, with half of the participants receiving collaborative and multimodal instruction, while the other half received traditional instruction. Pre and post-tests were conducted using the Oxford Quick Placement Test, and a modified version of the Language Learning Motivation and Engagement Scale was used to assess student engagement and motivation. Observational data were also collected to record instances of collaborative and multimodal activities. Quantitative analysis revealed that the experimental group exhibited a significant improvement in post-test scores and higher levels of engagement and motivation compared to the control group. The results suggest that the integration of collaborative and multimodal learning strategies can effectively enhance English language proficiency among ESL/EFL learners and contribute to the development of more inclusive and dynamic learning environments. The findings of this study contribute to the ongoing development of learner-centred approaches in ELT and offer valuable insights for educators seeking to enhance their teaching practices.

Decision Literacy, Multimodal Storytelling, and the Digital Storygame Project

A team of educators developed a multimodal learning strategy that focuses on the intersection of digital game design and traditional reading and writing skills.

Credit risk prediction of SMEs in supply chain finance by fusing demographic and behavioral data

The credit risk of small and medium-sized enterprises (SMEs) in supply chain finance (SCF) is defined as the probability that the SME would default on loans derived from financing for the SCF platform. Traditional models make use of merely the static data of SMEs, such as enterprise demographic data and financial statement data, to predict the credit risk of SMEs in SCF. Nevertheless, behavioral data, which reflect the dynamic financing behavior of SMEs in SCF, are overlooked by these models, which limits the performance of credit risk prediction. To address this problem, a novel approach is proposed called DeepRisk to fuse enterprise demographic data and financing behavioral data to predict the credit risk of SMEs in SCF. We adopt the multi-modal learning strategy to fuse the two different sources of data. The concatenated vectors derived from data fusion are then used as the input of the feed forward neural network to predict the credit risk of SMEs. Experiments on a real SCF dataset demonstrate that the proposed DeepRisk approach outperforms the baseline methods in credit risk prediction in terms of precision, recall, F1-score, area under curve (AUC), and economic loss. The fusion of the two different sources of data is superior to the existing approaches to the credit risk prediction of SMEs in SCF. Both the static enterprise demographic data and the dynamic financing behavioral data are crucial to improve the credit risk prediction of SMEs. Nevertheless, the variables derived from the financing behavioral data have a better predictability than those from the enterprise demographic data. Managerial implications have been identified for decision makers involved in SCF in utilizing the benefits of SCF and in managing their credit risks.

Multimodal learning strategy role in the formation of value orientations of the student youth

The purpose of this research is to study the exceptional nature of modern conditions for inculturation in higher education is, in particular, due to the introduction of a multimodal learning strategy into the educational process, focused on the training of broad specialists in terms of transitional economy. The paper presents a comparative analysis of the values of two generations of the student youth, formed in two fundamentally different systems of training highly qualified personnel. Modern methods of data analysis have made it possible to draw up a portrait of a modern student of higher education, including his/her needs and interests. The analysis showed that the use of the possibilities of a multimodal learning strategy contributed to the strengthening of individualism, a broad worldview were formed among young people. To improve the quality of training highly qualified personnel, it is proposed to shift the vector of a multimodal training strategy from solving urgent problems of economic markets to the mental needs of a person, which is a key link in the sustainable development of the economy.

Multimodality of the transmission of philological knowledge during classroom activities using ICT

The article deals with multimodal learning strategies aimed at transferring philological knowledge with the help of information and communications technology (ICT). Using ICT makes it possible to apply visual association and create spatial images that express important object properties defined clearly by the means of dynamic visualization (animation). The authors note that using ICT in the sphere is associated with several methodological difficulties, since it requires a philologist-educator who possess a wide range of competences that go beyond humanitarian knowledge. The optimum software that allows creating audio-visual construct (interactive poster, computer animation, tag cloud, cognitive map etc.) is offered for studying process. The paper presents the results of sociological research on the readiness of students of philological specialties in pedagogical universities along with the teachers in different educational units for implementation of multimodal strategies of teaching using ICT. The authors proved the need to introduce an integrative course “Use of ICT in the study of philological disciplines”.

Bifurcated Backbone Strategy for RGB-D Salient Object Detection.

Multi-level feature fusion is a fundamental topic in computer vision. It has been exploited to detect, segment and classify objects at various scales. When multi-level features meet multi-modal cues, the optimal feature aggregation and multi-modal learning strategy become a hot potato. In this paper, we leverage the inherent multi-modal and multi-level nature of RGB-D salient object detection to devise a novel Bifurcated Backbone Strategy Network (BBS-Net). Our architecture, is simple, efficient, and backbone-independent. In particular, first, we propose to regroup the multi-level features into teacher and student features using a bifurcated backbone strategy (BBS). Second, we introduce a depth-enhanced module (DEM) to excavate informative depth cues from the channel and spatial views. Then, RGB and depth modalities are fused in a complementary way. Extensive experiments show that BBS-Net significantly outperforms 18 state-of-the-art (SOTA) models on eight challenging datasets under five evaluation measures, demonstrating the superiority of our approach (~4% improvement in S-measure vs . the top-ranked model: DMRA). In addition, we provide a comprehensive analysis on the generalization ability of different RGB-D datasets and provide a powerful training set for future research. The complete algorithm, benchmark results, and post-processing toolbox are publicly available at https://github.com/zyjwuyan/BBS-Net.

Audio-Visual Deep Neural Network for Robust Person Verification

Voice and face are two most popular biometrics for person verification, usually used in speaker verification and face verification tasks. It has already been observed that simply combining the information from these two modalities can lead to a more powerful and robust person verification system. In this article, to fully explore the multi-modal learning strategies for person verification, we proposed three types of audio-visual deep neural network (AVN), including feature level AVN (AVN-F), embedding level AVN (AVN-E), and embedding level combination with joint learning AVN (AVN-J). To further enhance the system robustness in real noisy conditions where not both modalities can be accessed with high-quality, we proposed several data augmentation strategies for each proposed AVN: A feature-level multi-modal data augmentation is proposed for AVN-F and an embedding-level data augmentation with novel noise distribution matching is designed for AVN-E. For AVN-J, both the feature and embedding level multi-modal data augmentation methods can be applied. All the proposed models are trained on the VoxCeleb2 dev dataset and evaluated on the standard VoxCeleb1 dataset, and the best system achieves 0.558, 0.441% and 0.793% EER on the three official trial lists of VoxCeleb1, which is to our knowledge the best published single system results on this corpus for person verification. To validate the robustness of the proposed approaches, a noisy evaluation set based on the VoxCeleb1 is constructed, and experimental results show that the proposed system can significantly boost the system robustness and still show promising performance under this noisy scenario.

Implementing a Competency-Based Approach to Anatomy Teaching: Beginning With the End in Mind.

Background:The shift in the medical education system from a time-based to a competency-based model has encouraged its adoption and application in competency-based education in anatomy classrooms, such as team-based learning models and flipped classroom models. This pilot study aimed to build on previous work of the linkages between anatomy-based learning (a flipped classroom model inspired by a modified team-based learning) and student learning and engagement, and further to assess the linkage between anatomy-based learning and academic performance.Methods:A sequential mixed-methods design was employed to first gather and analyse quantitative data, including confidential student first semester scores in anatomy: gender, stream, anatomy-based learning, and final anatomy overall mark. The quantitative phase was followed by a qualitative phase in which a series of 8 anatomy laboratories were observed (4 anatomy-based learning and 4 traditional). Thematic analysis was performed on the observation data.Results:Aggregate anatomy-based learning and traditional stream tests, and final unit scores were compared. The anatomy-based learning and final unit scores showed little difference between students in the anatomy-based learning and students in the traditional stream. Students using anatomy-based learning had an aggregate score of 1.15 and final aggregate mark of 72, whereas students in the traditional section had an aggregate score of 1.19 and final mark of 79. Qualitative phase was undertaken to try to assess the linkages between anatomy-based learning and student learning. Observations showed that students in the anatomy-based learning section spent more time on task as compared with their peers in the traditional stream. The anatomy-based learning students also seemed to practice more self-directed learning and employed more multimodal learning strategies than the traditional section stream.Discussion/conclusions:Although the quantitative results of this study showed no significant difference in mean scores between anatomy-based learning and traditional designs, it was possible to observe the potential of flipped classroom model in engaging students in individual preparation, in team-based learning, and in consensus-based learning approaches.

Expanding treatment for opioid use disorder in publicly funded primary care clinics: Exploratory evaluation of the NYC health + hospitals buprenorphine ECHO program

Project Extension for Community Healthcare Outcomes (Project ECHO) offers an innovative and low-cost approach to enhancing the management of complex conditions among primary care providers. The NYC Health + Hospitals Buprenorphine ECHO (H + H ECHO) program offers primary care providers (PCPs) training and support in managing opioid use disorder (OUD). This exploratory study assessed the feasibility of a 16-session video conferencing platform led by Addiction Medicine experts in improving addiction knowledge, perceived self-efficacy, and buprenorphine prescribing among PCPs located in 17 publicly-funded ambulatory care clinics. A pre- and post-training survey assessed changes in knowledge and self-efficacy. Buprenorphine prescribing patterns were also captured pre-post training. Training sessions consisted of a review of the agenda by the H + H ECHO hub team, 15–30 min didactic lectures led by specialists, followed by a patient case presentation. Participants attended an average of 9 lectures (range, 1–15 sessions) and 53% of trainees attended at least 10 of the 16 sessions. Perceived self-efficacy improved post-H + H ECHO (73.2%) versus pre-training survey results (58.1%). There were minimal increases in knowledge post-training (58.4%) versus pre-training (51.4%). Only three additional providers reported prescribing Buprenorphine post-training (n = 10) versus pre-training (n = 7). Suggestions for improving H + H ECHO included trainings addressing stigma, administrative support, improved referrals to office-based opioid treatment (OBOT), integration of non-physician staff (i.e., case management, social work), and combining multimodal learning strategies (i.e., podcasts, web-based modules) with videoconferencing. This study demonstrates the feasibility of H + H ECHO among PCPs in publicly-funded clinics and improvements in self-efficacy. Studies are needed to identify alternative strategies to improve knowledge and prescribing of buprenorphine post-H + H ECHO.

Map learning and working memory: Multimodal learning strategies.

The current research investigated whether learning spatial information from a map involves different modalities, which are managed by discrete components in working memory. In four experiments, participants studied a map either while performing a simultaneous interference task (high cognitive load) or without interference (low cognitive load). The modality of interference varied between experiments. Experiment 1 used a tapping task (visuospatial), Experiment 2 a backward counting task (verbal), Experiment 3 an articulatory suppression task (verbal) and Experiment 4 an n-back task (central executive). Spatial recall was assessed in two tests: directional judgements and map drawing. Cognitive load was found to affect spatial recall detrimentally regardless of interference modality. The findings suggest that when learning maps, people use a multimodal learning strategy, utilising resources from all components of working memory.

Applying Knowledge Translation Concepts and Strategies in Dementia Care Education for Health Professionals

Dementia education programs are being developed for health professionals, but with limited guidance about "what works" in design and content to promote best practice in dementia care. Knowledge translation (KT) is a conceptual framework for putting evidence to work in health care. This narrative literature review examined the question: What does the field KT offer, conceptually and practically, for education of health professionals in dementia care? It seeks to identify the types of strategies currently used within education to facilitate effective KT for the wide range of health professionals who may be involved in the care of people with dementia, plus explore enablers and barriers to KT in this context. From 76 articles identified in academic databases and manual bibliographic searching, 22 met review criteria. The literature synthesis indicated four hallmarks of successful KT-oriented dementia education for health professionals: (1) multimodal delivery, (2) tailored approaches, (3) relationship building, and (4) organizational support for change in the work setting. Participatory action frameworks were also favored, based on interactive knowledge exchange (eg, blended learning) rather than passive unidirectional approaches alone (eg, lectures). The following six principles are proposed for educating health professionals in dementia care: (1) Match the education strategy to the KT goal and learner preferences; (2) Use integrated multimodal learning strategies and provide opportunities for multiple learning exposures plus feedback; (3) Build relationships to bridge the research-practice gap; (4) Use a simple compelling message with formats and technologies relevant to the audience; (5) Provide incentives to achieve KT goals; and (6) Plan to change the workplace, not just the individual health professional.

Lexical learning in mild aphasia: Gesture benefit depends on patholinguistic profile and lesion pattern

Gestures accompany speech and enrich human communication. When aphasia interferes with verbal abilities, gestures become even more relevant, compensating for and/or facilitating verbal communication. However, small-scale clinical studies yielded diverging results with regard to a therapeutic gesture benefit for lexical retrieval. Based on recent functional neuroimaging results, delineating a speech-gesture integration network for lexical learning in healthy adults, we hypothesized that the commonly observed variability may stem from differential patholinguistic profiles in turn depending on lesion pattern. Therefore we used a controlled novel word learning paradigm to probe the impact of gestures on lexical learning, in the lesioned language network.Fourteen patients with chronic left hemispheric lesions and mild residual aphasia learned 30 novel words for manipulable objects over four days. Half of the words were trained with gestures while the other half were trained purely verbally. For the gesture condition, rootwords were visually presented (e.g., Klavier, [piano]), followed by videos of the corresponding gestures and the auditory presentation of the novel words (e.g., /krulo/). Participants had to repeat pseudowords and simultaneously reproduce gestures. In the verbal condition no gesture-video was shown and participants only repeated pseudowords orally.Correlational analyses confirmed that gesture benefit depends on the patholinguistic profile: lesser lexico-semantic impairment correlated with better gesture-enhanced learning. Conversely largely preserved segmental-phonological capabilities correlated with better purely verbal learning. Moreover, structural MRI-analysis disclosed differential lesion patterns, most interestingly suggesting that integrity of the left anterior temporal pole predicted gesture benefit. Thus largely preserved semantic capabilities and relative integrity of a semantic integration network are prerequisites for successful use of the multimodal learning strategy, in which gestures may cause a deeper semantic rooting of the novel word-form. The results tap into theoretical accounts of gestures in lexical learning and suggest an explanation for the diverging effect in therapeutical studies advocating gestures in aphasia rehabilitation.

Sustainable Practice Improvements: Impact of the Comprehensive Advanced Palliative Care Education (CAPCE) Program

This paper describes an education program designed to improve palliative care practice through the development of workplace hospice palliative care resources (PCRs), and its impact on knowledge transfer and longer-term changes to clinical practice. Evaluation methods included pre- and post-program questionnaires, and a survey of learners' (n=301) perceptions of program learning strategies. Interviews (n=21) were conducted with a purposeful sample of PCRs and representatives from their work sites. Ratings of the sessions indicated that they were relevant to learners' clinical practice. At follow up, the majority of learners (83%) continued to serve as PCRs. Many positive effects were identified, including enhanced pain and symptom management, staff education, and development of care policies and guidelines. Management support, particularly the prioritization of palliative care and staff development, were factors facilitating sustained implementation. These findings highlight the importance of multimodal learning strategies and supportive work environments in the development of PCRs to enhance palliative care practice.