Multimodal Analysis Framework Research Articles

CANival: A multimodal approach to intrusion detection on the vehicle CAN bus

Vehicles of today are composed of over 100 electronic embedded devices known as Electronic Control Units (ECU), each of which controls a different component of the vehicle and communicates via the Controller Area Network (CAN) bus. However, unlike other network protocols, the CAN bus communication protocol lacks security features, which is a growing concern as more vehicles become connected to the Internet. To enable the detection of intrusions on the CAN bus, numerous intrusion detection systems (IDS) have been proposed. Although some are able to achieve high accuracy in detecting specific attacks, no IDS has been able to accurately detect all types of attacks against the CAN bus. To overcome the aforementioned issues, we propose a multimodal analysis framework named CANival, which consists of time interval-based and signal-based analyzers developed by designing a novel Time Interval Likelihood (TIL) model and optimizing an existing model CANet. Experimental results show that our multimodal IDS outperforms the base models and enhances the detection performance testing on two recent datasets, X-CANIDS Dataset and SynCAN, achieving average true positive rates of 0.960 and 0.912, and true negative rates of 0.997 and 0.996, respectively.

MOSAIC: An Artificial Intelligence-Based Framework for Multimodal Analysis, Classification, and Personalized Prognostic Assessment in Rare Cancers.

Rare cancers constitute over 20% of human neoplasms, often affecting patients with unmet medical needs. The development of effective classification and prognostication systems is crucial to improve the decision-making process and drive innovative treatment strategies. We have created and implemented MOSAIC, an artificial intelligence (AI)-based framework designed for multimodal analysis, classification, and personalized prognostic assessment in rare cancers. Clinical validation was performed on myelodysplastic syndrome (MDS), a rare hematologic cancer with clinical and genomic heterogeneities. We analyzed 4,427 patients with MDS divided into training and validation cohorts. Deep learning methods were applied to integrate and impute clinical/genomic features. Clustering was performed by combining Uniform Manifold Approximation and Projection for Dimension Reduction + Hierarchical Density-Based Spatial Clustering of Applications with Noise (UMAP + HDBSCAN) methods, compared with the conventional Hierarchical Dirichlet Process (HDP). Linear and AI-based nonlinear approaches were compared for survival prediction. Explainable AI (Shapley Additive Explanations approach [SHAP]) and federated learning were used to improve the interpretation and the performance of the clinical models, integrating them into distributed infrastructure. UMAP + HDBSCAN clustering obtained a more granular patient stratification, achieving a higher average silhouette coefficient (0.16) with respect to HDP (0.01) and higher balanced accuracy in cluster classification by Random Forest (92.7% ± 1.3% and 85.8% ± 0.8%). AI methods for survival prediction outperform conventional statistical techniques and the reference prognostic tool for MDS. Nonlinear Gradient Boosting Survival stands in the internal (Concordance-Index [C-Index], 0.77; SD, 0.01) and external validation (C-Index, 0.74; SD, 0.02). SHAP analysis revealed that similar features drove patients' subgroups and outcomes in both training and validation cohorts. Federated implementation improved the accuracy of developed models. MOSAIC provides an explainable and robust framework to optimize classification and prognostic assessment of rare cancers. AI-based approaches demonstrated superior accuracy in capturing genomic similarities and providing individual prognostic information compared with conventional statistical methods. Its federated implementation ensures broad clinical application, guaranteeing high performance and data protection.

Research on the Path of English Translation Teaching and External Communication Based on the Multimodal Analysis Method

Abstract With the development of the times and the continuous innovation of modern education technology, the multimodal theory is developing deeply in the field of education. The first step in this study is to use the optimization principle to select modes that meet the needs of English translation teaching and then develop a multimodal analysis framework for external communication that incorporates multimedia communication methods. Then, we constructed an emotion recognition model for English translation teaching by using speech recognition and expression recognition combined with convolutional neural networks for the students’ classroom situation and attitude and carried out the empirical analysis. The recognition accuracy of the emotion recognition system constructed in this paper is 86.4%. In the multimodal analysis of English translation teaching, the teacher who uses the linguistic modality type for the most number of times 387 times, and the teacher who uses the mixed modality for the most number of times 271 times, the students have the best classroom status. It has been stable at about 5 points, which is a “more positive” state. The students in the class had the highest level of motivation. Through the analysis, it is found that multimodal teaching can effectively improve the students’ status in class and keep their attention, which verifies the appropriateness of the external communication path combined with multimedia communication in multimodal English translation teaching. It provides a practical path for the development of English translation teaching.

Modeling and analyzing single-cell multimodal data with deep parametric inference.

The proliferation of single-cell multimodal sequencing technologies has enabled us to understand cellular heterogeneity with multiple views, providing novel and actionable biological insights into the disease-driving mechanisms. Here, we propose a comprehensive end-to-end single-cell multimodal analysis framework named Deep Parametric Inference (DPI). DPI transforms single-cell multimodal data into a multimodal parameter space by inferring individual modal parameters. Analysis of cord blood mononuclear cells (CBMC) reveals that the multimodal parameter space can characterize the heterogeneity of cells more comprehensively than individual modalities. Furthermore, comparisons with the state-of-the-art methods on multiple datasets show that DPI has superior performance. Additionally, DPI can reference and query cell types without batch effects. As a result, DPI can successfully analyze the progression of COVID-19 disease in peripheral blood mononuclear cells (PBMC). Notably, we further propose a cell state vector field and analyze the transformation pattern of bone marrow cells (BMC) states. In conclusion, DPI is a powerful single-cell multimodal analysis framework that can provide new biological insights into biomedical researchers. The python packages, datasets and user-friendly manuals of DPI are freely available at https://github.com/studentiz/dpi.

Real-Time Construction of Thermal Model Based on Multimodal Scene Data

In commercial buildings, the total consumption of central air conditioning accounts for about 40%–50%. However, at present, the initial design value of building Heating Ventilation and Air Conditioning (HVAC) is usually far greater than the actual refrigeration value of refrigeration demand, which will lead to great energy consumption waste. Moreover, the operation of HVAC affects the thermal comfort of users, so it is necessary to establish a thermal model for the scene to control. The thermal model describes the temperature of the scene in different environments. So it is very important to design a thermal model to calculate the scene in real time. Because the flow of people, the opening of windows, the ventilation of the scene and other parameters influence the change of thermal state in the scene environment, these parameters are complicated to model. Human disturbance will lead to the instability of the state of the scene environment. The inconsistency of its thermal model will lead to energy allocation tracking strategies in different regions. To solve this problem, We propose a thermal model for building thermal comfort using a multimodal analysis framework. This paper analyzes multiple temperature and humidity sensors and area image by multimodal combination and processes the image and sensor data by combining CNN and LSTM. Our results show that when the thermal model analyzed by this method is deployed in a building in the south of China, the MSE accuracy of the local effect of temperature field prediction reaches 99%, and its AMAX reaches 94%, so the running stability of the model in the scene is high. In addition, the research shows that the thermal model analysis framework can make the Internet of Things (IoT) in buildings more intelligent, and it can be combined with this thermal model to improve human comfort, make it easier to deploy in each hot zone, and have a better overall energy-saving effect.

“Because we have lived it”: Chicanx/Latinx Youth Multimodal Literacies in Youth Participatory Action Research

AbstractThis article highlights how Chicanx/Latinx youth draw on their lived realities as lenses to guide their inquiry and share their findings in a Youth Participatory Action Research project. In this project, youth collected testimonios from residents and staff within their migrant housing community to better understand histories of migration and community building. Drawing on a critical multimodal analysis framework which centers Chicanx/Latinx youth literacies and epistemologies, this article focuses on how youth drew on their community cultural knowledge and lived experiences in multimodal creation to bring attention to sociopolitical issues they cared about and impacted their community.

Codemeshing or Linguistic Purity: A Case Study of Wechat Languaging among Youngsters

What are the specific manifestations of the codemeshing? In Wechat, how is this performance further reflected? What is the modality frequently used by youngsters? What are the characteristics of this modality and how do they realize? How do youngsters treat language? These seemingly simple questions are actually a profound discussion of social and cultural phenomena. This study seeks to solve these problems. The case in this research comes from a group of 23-27-year-old postgraduates of English major, taking the currently widely used social media---WeChat as the research platform, using case studies as research method, and Vygotsky’s Social and Cultural Theory as well as the Visual Grammar under the multi-modal analysis framework as the theoretical support, with a combination of quantitative and qualitative analysis, tries to analyze codemeshing discourse, and conclude with the youngsters’ attitude towards language to explore language development issues.

How does digital context influence interaction in large live online lectures? The case of English-medium instruction

ABSTRACT This paper examines the evolving genre of university lectures. It focuses on synchronous online lectures. The aim of the study is to shed some light on how interaction between teacher and students unfolds in large English-medium instruction (EMI) lectures in the digital context. A qualitative multimodal microanalysis of an episode of interaction was performed from an (inter)action multimodal analysis framework. This preliminary exploratory study reveals the structural and multimodal complexity of interaction in live online lectures. EMI teacher’s semiotic resources combine to make meaning comprehensible in a lingua franca and to engage learners in a virtual context where there is not eye-contact with them. Suggestions are made to undertake contrastive studies on interaction in online and face-to-face lectures that may respond to the need of EMI teacher training. This paper aims to contribute to the literature of this still unexplored academic instructional digital genre.

Multimodal cooperative learning for micro-video advertising click prediction

PurposeMicro-video platforms have gained attention in recent years and have also become an important new channel for merchants to advertise their products. Since little research has studied micro-video advertising, this paper aims to fill the research gap by exploring the determinants of micro-video advertising clicks. We form a micro-video advertising click prediction model and demonstrate the effectiveness of the multimodal information extracted from the advertisement producers, commodities being sold and micro-video contents in the prediction task.Design/methodology/approachA multimodal analysis framework was conducted based on real-world micro-video advertisement datasets. To better capture the relations between different modalities, we adopt a cooperative learning model to predict the advertising clicks.FindingsThe experimental results show that the features extracted from different data sources can improve the prediction performance. Furthermore, the combination of different modal features (visual, acoustic, textual and numerical) is also worth studying. Compared to classical baseline models, the proposed cooperative learning model significantly outperforms the prediction results, which demonstrates that the relations between modalities are also important in advertising micro-video generation.Originality/valueTo the best of our knowledge, this is the first study analysing micro-video advertising effects. With the help of our advertising click prediction model, advertisement producers (merchants or their partners) can benefit from generating more effective micro-video advertisements. Furthermore, micro-video platforms can apply our prediction results to optimise their advertisement allocation algorithm and better manage network traffic. This research can be of great help for more effective development of the micro-video advertisement industry.

Harnessing multimodality in language teacher education: expanding English-dominant teachers’ translanguaging capacities through a Multimodalities Entextualization Cycle

ABSTRACT This qualitative case study (Yin, [2014]. Case Study Research: Design and Methods. 5th Ed. Thousand Oaks, CA: Sage.) emerged from our effort to support teachers, who predominantly identify as monolingual speakers, in learning about translanguaging through a less-explored facet of translanguaging: multimodality. Drawing upon Bezemer and Kress’s (2016) social semiotic framework for multimodal analysis and Lin’s, [2019]. “Theories of Trans/Languaging and Trans-Semiotizing: Implications for Content-Based Education Classrooms.” International Journal of Bilingual Education and Bilingualism 22 (1): 5–16. doi:10.1080/13670050.2018.1515175.) Multimodalities Entextualization Cycle (MEC), we ask: ‘What did integration of multimodal resources contribute to pre- and in-service teachers’ learning about translanguaging in an online asynchronous course within an English as a Second Language teacher education program at a Midwest U.S. university?’ Through multimodal analysis (Jewitt 2017), we analyzed the texts teachers created in response to four multimodal tasks; this included analyzing semiotic meaning conveyed through the multiple modes teachers integrated in their responses. We also compared broader patterns in their cumulative learning about translanguaging as a part of an MEC. Our findings suggest that envisioning a cycle of multimodal learning tasks can provide predominantly monolingual-identifying teachers with means to expand their own dynamic repertoires as they make sense of translanguaging. We conclude with recommendations for how language teacher educators can harness multimodal resources to expand teachers’ capacity to enact translanguaging as an equity-based pedagogy of hope. (194 words)

An Efficient WRF Framework for Discovering Risk Genes and Abnormal Brain Regions in Parkinson’s Disease Based on Imaging Genetics Data

As an emerging research field of brain science, multimodal data fusion analysis has attracted broader attention in the study of complex brain diseases such as Parkinson’s disease (PD). However, current studies primarily lie with detecting the association among different modal data and reducing data attributes. The data mining method after fusion and the overall analysis framework are neglected. In this study, we propose a weighted random forest (WRF) model as the feature screening classifier. The interactions between genes and brain regions are detected as input multimodal fusion features by the correlation analysis method. We implement sample classification and optimal feature selection based on WRF, and construct a multimodal analysis framework for exploring the pathogenic factors of PD. The experimental results in Parkinson’s Progression Markers Initiative (PPMI) database show that WRF performs better compared with some advanced methods, and the brain regions and genes related to PD are detected. The fusion of multi-modal data can improve the classification of PD patients and detect the pathogenic factors more comprehensively, which provides a novel perspective for the diagnosis and research of PD. We also show the great potential of WRF to perform the multimodal data fusion analysis of other brain diseases.

Film, dialects and subtitles: an analytical framework for the study of non-standard varieties in subtitling

ABSTRACTIn this article we will propose a framework of multimodal analysis for the study of the translation of linguistic varieties in subtitled audiovisual products. Three dimensions are suggested – textual, diegetic and sociocultural. This will offer tools to identify: (a) the linguistic varieties; (b) how they were recreated in both the source text and the target text; (c) the communicative function they fulfil and how they are multimodally erected; (d) the impact of specific translation strategies and, finally, (e) the possible social mediating factors behind the translator’s decisions. We hope to highlight the need to go beyond the analysis of the spoken mode and to examine the impact of the subtitling strategies both in their multimodal and broader filmic and sociocultural contexts.

Towards an intelligent framework for multimodal affective data analysis

An increasingly large amount of multimodal content is posted on social media websites such as YouTube and Facebook everyday. In order to cope with the growth of such so much multimodal data, there is an urgent need to develop an intelligent multi-modal analysis framework that can effectively extract information from multiple modalities. In this paper, we propose a novel multimodal information extraction agent, which infers and aggregates the semantic and affective information associated with user-generated multimodal data in contexts such as e-learning, e-health, automatic video content tagging and human–computer interaction. In particular, the developed intelligent agent adopts an ensemble feature extraction approach by exploiting the joint use of tri-modal (text, audio and video) features to enhance the multimodal information extraction process. In preliminary experiments using the eNTERFACE dataset, our proposed multi-modal system is shown to achieve an accuracy of 87.95%, outperforming the best state-of-the-art system by more than 10%, or in relative terms, a 56% reduction in error rate.

A Multimodal Analysis of Image-text Relations in Picture Books

Image-text relations are expounded based on Systemic Functional Multimodal Analysis framework in this article. A though analysis of the logical relations between the visual and verbal elements in picture books is also given in terms of elaboration, extension, enhancement and projection. Furthermore, Divergence, a fundamental type of image-text relation in picture books, is thoroughly investigated. Generic framework of picture book is proposed for the exploration of image-text relations.

A Three-Step Multimodal Analysis Framework for Modeling Corticomuscular Activity With Application to Parkinson’s Disease

Corticomuscular coupling analysis based on multiple datasets such as electroencephalography (EEG) and electromyography (EMG) signals provides a useful tool for understanding human motor control systems. A popular conventional method to assess corticomuscular coupling has been the pair-wise magnitude-squared coherence (MSC) between EEG and concomitant EMG recordings. However, there are certain limitations associated with the MSC, including the difficulty in robustly assessing group inference, only dealing with two types of datasets simultaneously and the biologically implausible assumption of pair-wise interactions. To overcome such limitations, in this paper, we propose assessing corticomuscular coupling by combining multiset canonical correlation analysis (M-CCA) and joint independent component analysis (jICA). The proposed method takes advantage of the M-CCA and jICA to ensure that the extracted components are maximally correlated across multiple datasets and meanwhile statistically independent within each dataset. Simulations were performed to illustrate the performance of the proposed method. We also applied the proposed method to concurrent EEG, EMG, and behavior data collected in a Parkinson's disease (PD) study. The results reveal highly correlated temporal patterns among the three types of signals and corresponding spatial activation patterns. In addition to the expected motor areas, the corresponding spatial activation patterns demonstrate enhanced occipital connectivity in the PD subjects, consistent with previous medical findings.

A Joint Multimodal Group Analysis Framework for Modeling Corticomuscular Activity

Corticomuscular coupling analysis based on multiple data sets such as electroencephalography (EEG) and electromyography (EMG) signals provides a useful tool for understanding human motor control systems. Two probably most popular methods are the pair-wise magnitude-squared coherence (MSC) between EEG and simultaneously-recorded EMG signals, and partial least square (PLS). Unfortunately, MSC and PLS generally deal with only two types of data sets at the same time, while we may need to analyze more than two types of data sets. Moreover, it is not straightforward to extend MSC to the group level for combining results across subjects. Also, PLS can have the information mixing problem since only the variations in one data set are used to predict the other data set. To address these concerns, we propose a joint multimodal analysis framework for corticomuscular coupling analysis. The proposed framework models multiple data spaces simultaneously in a multidirectional fashion. Furthermore, to address the inter-subject variability concern in real-world medical applications, we extend the proposed framework from the individual subject level to the group level to obtain common corticomuscular coupling patterns across subjects. We apply the proposed framework to concurrent EEG, EMG and behavior data collected in a Parkinson's disease (PD) study. The results reveal several highly correlated temporal patterns among the three types of signals and their corresponding spatial activation patterns. In PD subjects, there are enhanced connections between occipital region and other regions, which is consistent with the previous medical finding. The proposed framework is a promising technique for performing multi-subject and multi-modal data analysis.

Bringing the World to School: Integrating News and Media Literacy in Elementary Classrooms

This qualitative study explores how adolescent high school students in an AP English class used multiple forms of media (the Internet, digital video, slide show software, video editing tools, literary texts, and writing) to respond to and analyze a contemporary novel, The Kite Runner, by Khaled Hosseini. Using a multimodal analysis framework, the author explores the following question: How are students’ multimodal retellings with literary devices mediated by the text, the choice of compositional tool, and the use of different modes? The findings of this study can be used to shape our understanding of adolescents’ multimodal composition practices in academic settings.

An Interactive Device for Quick Arabic News Story Browsing

The authors propose a framework for multimodal analysis of Arabic news broadcast which helps users of pervasive devices to browsing quickly into news archive; their solution integrating many aspects such as summarizing, indexing textual content and on on-line recognition of the handwriting. Firstly, the summarizing process is to accelerate the video content browsing based on genetic algorithm. Secondly, the indexing process, which operates on video summaries based on text recognition. Finally users communicate by writing keywords on PDA screen and keep only summaries speaking about this topic. This PDA contains an on line recognition system of Arabic of handwritten based on visual coding and genetic algorithm.

Robust classification of neonatal apnoea-related desaturations

Respiratory signals monitored in the neonatal intensive care units are usually ignored due to the high prevalence of noise and false alarms (FA). Apneic events are generally therefore indicated by a pulse oximeter alarm reacting to the subsequent desaturation. However, the high FA rate in the photoplethysmogram may desensitize staff, reducing the reaction speed. The main reason for the high FA rates of critical care monitors is the unimodal analysis behaviour. In this work, we propose a multimodal analysis framework to reduce the FA rate in neonatal apnoea monitoring. Information about oxygen saturation, heart rate, respiratory rate and signal quality was extracted from electrocardiogram, impedance pneumogram and photoplethysmographic signals for a total of 20 features in the 5 min interval before a desaturation event. 1616 desaturation events from 27 neonatal admissions were annotated by two independent reviewers as true (physiologically relevant) or false (noise-related). Patients were divided into two independent groups for training and validation, and a support vector machine was trained to classify the events as true or false. The best classification performance was achieved on a combination of 13 features with sensitivity, specificity and accuracy of 100% in the training set, and a sensitivity of 86%, a specificity of 91% and an accuracy of 90% in the validation set.

Guest Editors' Introduction: Special Section on Connected Multimedia

Social media has received extensive attention recently and has become a very popular research area due to its wide spectrum of applications. We note that even though the whole area of social media is very popular in the literature, there are a group of research issues that are related to the social-cultural constraints in the social media study that have not yet received sufficient attention. In this context, we group all these issues together under the umbrella of a new sub-area of social media that we call connected multimedia . Consequently, by connected multimedia, we mean the study of the social and technical interactions among users, multimedia data, and devices across cultures and the explicit exploitation of cultural differences. Hence, connected multimedia involves the three elements – the users, the multimedia data, and the devices – with two perspectives – the social focus and the cultural focus. In short, connected multimedia is about multimedia content and connection across community and cultural boundaries. In comparison with those existing research areas including social media as its super-area and human centered computing, we here emphasize that connected multimedia pays more attention to the cultural difference. The definition of the social side is broader than just national cultures; it possibly includes cultures of groups, disciplines, organizations, communities, ethnicities, religions, and nations. This emphasis distinguishes connected multimedia from all other existing areas such as social media and cross-media, which may either claim to include some of these aspects, among many others, or have different foci. This special section is organized following the successful first two editions of the workshop on the newly emerged theme of connected multimedia held in Hangzhou, China, in October of 2009 and in Florence, Italy, in October, 2010. After two rounds of rigorous reviews for all the submissions in response to the CFP of this special section, we have finally selected three papers to be included in this special section on connected multimedia. The paper “Association of Moving Objects across Visual Sensor Networks” by M.J. Mirza and N. Anjum presents an inter-camera trajectory association algorithm for partially overlapping visual sensor networks; the paper addresses issues related to trajectory extraction, representation, and association. The paper “Faceted Subtopic Retrieval: Exploiting the Topic Hierarchy via a Multi-Modal Framework” by J. Sang and C. Xu presents a new framework for multi-modal analysis and retrieval called faceted subtopic retrieval that is tailored to complex queries to the social media data on the Web concerning political and social events or issues. A LDA-like model is developed to exploit the intrinsic topic hierarchy inside the retrieved data. The paper “Tour Route Recommendation Begins with Multimodal Classification” by X. Chen and Q. Wang addresses the problem of location estimation for tourist photos; the paper proposes a solution that begins with classification and exploits explicitly the textual, temporal, geographic, and visual information together for a tour route recommendation. Overall, all the three papers address the theory and application case studies on the general topic of connected multimedia. While this editorial is co-authored by the guest-editors who also organized the two editions of the workshop on the same topic that actually led to this special section, many people have contributed the ideas that have finally led to the development of this topic of connected multimedia in the literature. Specifically, Noshir Contractor, Alan Hanjalic, Alexander G. Hauptmann, Xian-Sheng Hua, Alejandro (Alex) Jaimes, Ivan Ivanov, Michael S. Lew, Wanqing Li, Ching-Yung Ling, Alexander C. Loui, Jiebo Luo, Michael W. Macy, Nicu Sebe, Qi Tian, Yonghong Tian, Vincent S. Tseng, Qing Wang, Changsheng Xu, Huimin Yu, and Shiwen Yu deserve the credit for contributing their ideas to the development of the literature on this topic. Finally, we would like to thank Dr. Jie Yang, US NSF Program Manager, for the support to the development of this effort and Dr. Jiebo Luo, the EIC of Journal of Multimedia, for the support to publishing this special section. We also acknowledge US NSF (through grant IIS-0956924), Zhejiang University College of Computer Science and Technology, Microsoft Research, and ACM for sponsoring the two editions of the workshop on connected multimedia that led to this special section. This effort of editing this special section is also supported in part by US NSF (IIS-0812114, CCF-1017828) and National Basic Research Program of China (2012CB316400).