Multimodal Deep Learning Architecture Research Articles

The EarSAVAS Dataset

Subject-aware vocal activity sensing on wearables, which specifically recognizes and monitors the wearer's distinct vocal activities, is essential in advancing personal health monitoring and enabling context-aware applications. While recent advancements in earables present new opportunities, the absence of relevant datasets and effective methods remains a significant challenge. In this paper, we introduce EarSAVAS, the first publicly available dataset constructed specifically for subject-aware human vocal activity sensing on earables. EarSAVAS encompasses eight distinct vocal activities from both the earphone wearer and bystanders, including synchronous two-channel audio and motion data collected from 42 participants totaling 44.5 hours. Further, we propose EarVAS, a lightweight multi-modal deep learning architecture that enables efficient subject-aware vocal activity recognition on earables. To validate the reliability of EarSAVAS and the efficiency of EarVAS, we implemented two advanced benchmark models. Evaluation results on EarSAVAS reveal EarVAS's effectiveness with an accuracy of 90.84% and a Macro-AUC of 89.03%. Comprehensive ablation experiments were conducted on benchmark models and demonstrated the effectiveness of feedback microphone audio and highlighted the potential value of sensor fusion in subject-aware vocal activity sensing on earables. We hope that the proposed EarSAVAS and benchmark models can inspire other researchers to further explore efficient subject-aware human vocal activity sensing on earables.

Ex-Vivo Hippocampus Segmentation Using Diffusion-Weighted MRI

The hippocampus is a crucial brain structure involved in memory formation, spatial navigation, emotional regulation, and learning. An accurate MRI image segmentation of the human hippocampus plays an important role in multiple neuro-imaging research and clinical practice, such as diagnosing neurological diseases and guiding surgical interventions. While most hippocampus segmentation studies focus on using T1-weighted or T2-weighted MRI scans, we explore the use of diffusion-weighted MRI (dMRI), which offers unique insights into the microstructural properties of the hippocampus. Particularly, we utilize various anisotropy measures derived from diffusion MRI (dMRI), including fractional anisotropy, mean diffusivity, axial diffusivity, and radial diffusivity, for a multi-contrast deep learning approach to hippocampus segmentation. To exploit the unique benefits offered by various contrasts in dMRI images for accurate hippocampus segmentation, we introduce an innovative multimodal deep learning architecture integrating cross-attention mechanisms. Our proposed framework comprises a multi-head encoder designed to transform each contrast of dMRI images into distinct latent spaces, generating separate image feature maps. Subsequently, we employ a gated cross-attention unit following the encoder, which facilitates the creation of attention maps between every pair of image contrasts. These attention maps serve to enrich the feature maps, thereby enhancing their effectiveness for the segmentation task. In the final stage, a decoder is employed to produce segmentation predictions utilizing the attention-enhanced feature maps. The experimental outcomes demonstrate the efficacy of our framework in hippocampus segmentation and highlight the benefits of using multi-contrast images over single-contrast images in diffusion MRI image segmentation.

A multimodal deep learning architecture for smoking detection with a small data approach.

Covert tobacco advertisements often raise regulatory measures. This paper presents that artificial intelligence, particularly deep learning, has great potential for detecting hidden advertising and allows unbiased, reproducible, and fair quantification of tobacco-related media content. We propose an integrated text and image processing model based on deep learning, generative methods, and human reinforcement, which can detect smoking cases in both textual and visual formats, even with little available training data. Our model can achieve 74% accuracy for images and 98% for text. Furthermore, our system integrates the possibility of expert intervention in the form of human reinforcement. Using the pre-trained multimodal, image, and text processing models available through deep learning makes it possible to detect smoking in different media even with few training data.

Pesti-DGI-Net: A multi-modal deep learning architecture based on dual interpretability for pesticide-likeness prediction

“Pesticide-likeness” represents an extension of the pharmaceutical concept of “drug-likeness” to the field of pesticides. The development of algorithmic tools for predicting pesticide-likeness holds great significance for the rational design of pesticide molecules. Among various computational approaches, artificial intelligence techniques, especially deep learning, stand out due to their distinctive advantages. However, the application of deep learning models in the field of pesticide-likeness remains relatively limited. To address this gap, we proposed a multi-modal deep learning architecture, termed Pesti-DGI-Net, which took the standard Simplified Molecular Input Line Entry System (SMILES) of compounds as input and combined molecular representations across multiple dimensions. Through this fusion, Pesti-DGI-Net made accurate predictions of the pesticide-likeness for candidate compounds, as substantiated by extensive evaluations on internal test sets and an external independent test set. Additionally, Pesti-DGI-Net provided two interpretable methods to elucidate the relationship between chemical structure and pesticide-likeness. Comparison with domain experts showed that Pesti-DGI-Net enabled researchers to better understand the prediction results. Finally, we integrated Pesti-DGI-Net with existing web resources to comprehensively assess the potential of compounds as pesticide-like molecules. Our cloud platform is freely available at http://chemyang.ccnu.edu.cn/ccb/server/CoPLE/.

Deep Learning for Technical Document Classification

In large technology companies, the requirements for managing and organizing technical documents created by engineers and managers have increased dramatically in recent years, which has led to a higher demand for more scalable, accurate, and automated document classification. Prior studies have only focused on processing text for classification, whereas technical documents often contain multimodal information. To leverage multimodal information for document classification to improve the model performance, this paper presents a novel multimodal deep learning architecture, TechDoc, which utilizes three types of information, including natural language texts and descriptive images within documents and the associations among the documents. The architecture synthesizes the convolutional neural network, recurrent neural network, and graph neural network through an integrated training process. We applied the architecture to a large multimodal technical document database and trained the model for classifying documents based on the hierarchical International Patent Classification system. Our results show that TechDoc presents a greater classification accuracy than the unimodal methods and other state-of-the-art benchmarks. The trained model can potentially be scaled to millions of real-world multimodal technical documents, which is useful for data and knowledge management in large technology companies and organizations.

Investigating permafrost carbon dynamics in Alaska with artificial intelligence

Positive feedbacks between permafrost degradation and the release of soil carbon into the atmosphere impact land–atmosphere interactions, disrupt the global carbon cycle, and accelerate climate change. The widespread distribution of thawing permafrost is causing a cascade of geophysical and biochemical disturbances with global impacts. Currently, few earth system models account for permafrost carbon feedback (PCF) mechanisms. This research study integrates artificial intelligence (AI) tools and information derived from field-scale surveys across the tundra and boreal landscapes in Alaska. We identify and interpret the permafrost carbon cycling links and feedback sensitivities with GeoCryoAI, a hybridized multimodal deep learning (DL) architecture of stacked convolutionally layered, memory-encoded recurrent neural networks (NN). This framework integrates in-situ measurements and flux tower observations for teacher forcing and model training. Preliminary experiments to quantify, validate, and forecast permafrost degradation and carbon efflux across Alaska demonstrate the fidelity of this data-driven architecture. More specifically, GeoCryoAI logs the ecological memory and effectively learns covariate dynamics while demonstrating an aptitude to simulate and forecast PCF dynamics—active layer thickness (ALT), carbon dioxide flux (CO2), and methane flux (CH4)—with high precision and minimal loss (i.e. ALTRMSE: 1.327 cm [1969–2022]; CO2 RMSE: 0.697 µmolCO2m−2s−1 [2003–2021]; CH4 RMSE: 0.715 nmolCH4m−2s−1 [2011–2022]). ALT variability is a sensitive harbinger of change, a unique signal characterizing the PCF, and our model is the first characterization of these dynamics across space and time.

Revealing the impact of psychiatric comorbidities on treatment outcome in early psychosis using counterfactual model explanation.

Psychiatric comorbidities have a significant impact on the course of illness in patients with schizophrenia spectrum disorders. To accurately predict outcomes for individual patients using computerized prognostic models, it is essential to consider these comorbidities and their influence. In our study, we utilized a multi-modal deep learning architecture to forecast symptomatic remission, focusing on a multicenter sample of patients with first-episode psychosis from the OPTiMiSE study. Additionally, we introduced a counterfactual model explanation technique to examine how scores on the Mini International Neuropsychiatric Interview (MINI) affected the likelihood of remission, both at the group level and for individual patients. Our findings at the group level revealed that most comorbidities had a negative association with remission. Among them, current and recurrent depressive disorders consistently exerted the greatest negative impact on the probability of remission across patients. However, we made an interesting observation: current suicidality within the past month and substance abuse within the past 12 months were associated with an increased chance of remission in patients. We found a high degree of variability among patients at the individual level. Through hierarchical clustering analysis, we identified two subgroups: one in which comorbidities had a relatively limited effect on remission (approximately 45% of patients), and another in which comorbidities more strongly influenced remission. By incorporating comorbidities into individualized prognostic prediction models, we determined which specific comorbidities had the greatest impact on remission at both the group level and for individual patients. These results highlight the importance of identifying and including relevant comorbidities in prediction models, providing valuable insights for improving the treatment and prognosis of patients with psychotic disorders. Furthermore, they open avenues for further research into the efficacy of treating these comorbidities to enhance overall patient outcomes.

Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems

User trust has been identified as a critical issue that is pivotal to the success of autonomous vehicle (AV) operations where artificial intelligence (AI) is widely adopted. For such integrated AI-based driving systems, one promising way of building user trust is through the concept of explainable artificial intelligence (XAI) which requires the AI system to provide the user with the explanations behind each decision it makes. Motivated by both the need to enhance user trust and the promise of novel XAI technology in addressing such need, this paper seeks to enhance trustworthiness in autonomous driving systems through the development of explainable Deep Learning (DL) models. First, the paper casts the decision-making process of the AV system not as a classification task (which is the traditional process) but rather as an image-based language generation (image captioning) task. As such, the proposed approach makes driving decisions by first generating textual descriptions of the driving scenarios, which serve as explanations that humans can understand. To this end, a novel multi-modal DL architecture is proposed to jointly model the correlation between an image (driving scenario) and language (descriptions). It adopts a fully Transformer-based structure and therefore has the potential to perform global attention and imitate effectively, the learning processes of human drivers. The results suggest that the proposed model can and does generate legal and meaningful sentences to describe a given driving scenario, and subsequently to correctly generate appropriate driving decisions in autonomous vehicles (AVs). It is also observed that the proposed model significantly outperforms multiple baseline models in terms of generating both explanations and driving actions. From the end user’s perspective, the proposed model can be beneficial in enhancing user trust because it provides the rationale behind an AV’s actions. From the AV developer’s perspective, the explanations from this explainable system could serve as a “debugging” tool to detect potential weaknesses in the existing system and identify specific directions for improvement.

Multimodal Approach for Enhancing Biometric Authentication.

Unimodal biometric systems rely on a single source or unique individual biological trait for measurement and examination. Fingerprint-based biometric systems are the most common, but they are vulnerable to presentation attacks or spoofing when a fake fingerprint is presented to the sensor. To address this issue, we propose an enhanced biometric system based on a multimodal approach using two types of biological traits. We propose to combine fingerprint and Electrocardiogram (ECG) signals to mitigate spoofing attacks. Specifically, we design a multimodal deep learning architecture that accepts fingerprints and ECG as inputs and fuses the feature vectors using stacking and channel-wise approaches. The feature extraction backbone of the architecture is based on data-efficient transformers. The experimental results demonstrate the promising capabilities of the proposed approach in enhancing the robustness of the system to presentation attacks.

A Deep Learning-Based Multimodal Architecture to predict Signs of Dementia

This paper proposes a multimodal deep learning architecture combining text and audio information to predict dementia, a disease which affects around 55 million people all over the world and makes them in some cases dependent people. The system was evaluated on the DementiaBank Pitt Corpus dataset, which includes audio recordings as well as their transcriptions for healthy people and people with dementia. Different models have been used and tested, including Convolutional Neural Networks (CNN) for audio classification, Transformers for text classification, and a combination of both in a multimodal ensemble. These models have been evaluated on a test set, obtaining the best results by using the text modality, achieving 90.36% accuracy on the task of detecting dementia. Additionally, an analysis of the corpus has been conducted for the sake of explainability, aiming to obtain more information about how the models generate their predictions and identify patterns in the data.

Multi-modal deep learning improves grain yield prediction in wheat breeding by fusing genomics and phenomics.

Developing new crop varieties with superior performance is highly important to ensure robust and sustainable global food security. The speed of variety development is limited by long field cycles and advanced generation selections in plant breeding programs. While methods to predict yield from genotype or phenotype data have been proposed, improved performance and integrated models are needed. We propose a machine learning model that leverages both genotype and phenotype measurements by fusing genetic variants with multiple data sources collected by unmanned aerial systems. We use a deep multiple instance learning framework with an attention mechanism that sheds light on the importance given to each input during prediction, enhancing interpretability. Our model reaches 0.754 ± 0.024 Pearson correlation coefficient when predicting yield in similar environmental conditions; a 34.8% improvement over the genotype-only linear baseline (0.559 ± 0.050). We further predict yield on new lines in an unseen environment using only genotypes, obtaining a prediction accuracy of 0.386 ± 0.010, a 13.5% improvement over the linear baseline. Our multi-modal deep learning architecture efficiently accounts for plant health and environment, distilling the genetic contribution and providing excellent predictions. Yield prediction algorithms leveraging phenotypic observations during training therefore promise to improve breeding programs, ultimately speeding up delivery of improved varieties. Available at https://github.com/BorgwardtLab/PheGeMIL (code) and https://doi.org/doi:10.5061/dryad.kprr4xh5p (data).

Towards a multi-modal Deep Learning Architecture for User Modeling

Deep learning has succeeded in various applications, including image classification and feature learning. However, there needs to be more research on its use in Intelligent Tutoring Systems or Serious Games, particularly in modeling user behavior during learning or gaming sessions using multi-modal data. Creating an effective user model is crucial for developing a highly adaptive system. To achieve this, it is necessary to consider all available data sources to inform the user’s current state. This study proposes a user-sensitive deep multi-modal architecture that leverages deep learning and user data to extract a rich latent representation of the user. The architecture combines a Long Short-Term Memory, a Convolutional Neural Network, and multiple Deep Neu-ral Networks to handle the multi-modality of data. The resulting model was evaluated on a public multi-modal dataset, achieving better results than state-of-the-art algorithms for a similar task: opinion polarity detection. These findings suggest that the latent representation learned from the data is useful in discriminating behaviors. This proposed solution can be applied in various contexts where user modeling using multi-modal data is critical for improving the user experience.

Integrating multimodal information in machine learning for classifying acute myocardial infarction

Objective. Prompt identification and recognization of myocardial ischemia/infarction (MI) is the most important goal in the management of acute coronary syndrome. The 12-lead electrocardiogram (ECG) is widely used as the initial screening tool for patients with chest pain but its diagnostic accuracy remains limited. There is early evidence that machine learning (ML) algorithms applied to ECG waveforms can improve performance. Most studies are designed to classify MI from healthy controls and thus are limited due to the lack of consideration of ECG abnormalities from other cardiac conditions, leading to false positives. Moreover, clinical information beyond ECG has not yet been well leveraged in existing ML models. Approach. The present study considered downstream clinical implementation scenarios in the initial model design by dichotomizing study recordings from a public large-scale ECG dataset into a MI class and a non-MI class with the inclusion of MI-confounding conditions. Two experiments were conducted to systematically investigate the impact of two important factors entrained in the modeling process, including the duration of ECG, and the value of multimodal information for model training. A novel multimodal deep learning architecture was proposed to learn joint features from both ECG and patient demographics. Main results. The multimodal model achieved better performance than the ECG-only model, with a mean area under the receiver operating characteristic curve of 92.1% and a mean accuracy of 87.4%, which is on par with existing studies despite the increased task difficulty due to the new class definition. By investigation of model explainability, it revealed the contribution of patient information in model performance and clinical concordance of the model’s attention with existing clinical insights. Significance. The findings in this study help guide the development of ML solutions for prompt MI detection and move the models one step closer to real-world clinical applications.

Intelligent antepartum fetal monitoring via deep learning and fusion of cardiotocographic signals and clinical data.

Cardiotocography (CTG), which measures uterine contraction (UC) and fetal heart rate (FHR), is a crucial tool for assessing fetal health during pregnancy. However, traditional computerized cardiotocography (cCTG) approaches have non-negligible calibration errors in feature extraction and heavily rely on the expertise and prior experience to define diagnostic features from CTG or FHR signals. Although previous works have studied deep learning methods for extracting CTG or FHR features, these methods still neglect the clinical information of pregnant women. In this paper, we proposed a multimodal deep learning architecture (MMDLA) for intelligent antepartum fetal monitoring that is capable of performing automatic CTG feature extraction, fusion with clinical data and classification. The multimodal feature fusion was achieved by concatenating high-level CTG features, which were extracted from preprocessed CTG signals via a convolution neural network (CNN) with six convolution layers and five fully connected layers, and the clinical data of pregnant women. Eventually, light gradient boosting machine (LGBM) was implemented as fetal status assessment classifier. The effectiveness of MMDLA was evaluated using a dataset of 16,355 cases, each of which includes FHR signal, UC signal and pertinent clinical data like maternal age and gestational age. With an accuracy of 90.77% and an area under the curve (AUC) value of 0.9201, the multimodal features performed admirably. The data imbalance issue was also effectively resolved by the LGBM classifier, with a normal-F1 value of 0.9376 and an abnormal-F1 value of 0.8223. In summary, the proposed MMDLA is conducive to the realization of intelligent antepartum fetal monitoring.

Prostate cancer therapy personalization via multi-modal deep learning on randomized phase III clinical trials

Prostate cancer is the most frequent cancer in men and a leading cause of cancer death. Determining a patient’s optimal therapy is a challenge, where oncologists must select a therapy with the highest likelihood of success and the lowest likelihood of toxicity. International standards for prognostication rely on non-specific and semi-quantitative tools, commonly leading to over- and under-treatment. Tissue-based molecular biomarkers have attempted to address this, but most have limited validation in prospective randomized trials and expensive processing costs, posing substantial barriers to widespread adoption. There remains a significant need for accurate and scalable tools to support therapy personalization. Here we demonstrate prostate cancer therapy personalization by predicting long-term, clinically relevant outcomes using a multimodal deep learning architecture and train models using clinical data and digital histopathology from prostate biopsies. We train and validate models using five phase III randomized trials conducted across hundreds of clinical centers. Histopathological data was available for 5654 of 7764 randomized patients (71%) with a median follow-up of 11.4 years. Compared to the most common risk-stratification tool—risk groups developed by the National Cancer Center Network (NCCN)—our models have superior discriminatory performance across all endpoints, ranging from 9.2% to 14.6% relative improvement in a held-out validation set. This artificial intelligence-based tool improves prognostication over standard tools and allows oncologists to computationally predict the likeliest outcomes of specific patients to determine optimal treatment. Outfitted with digital scanners and internet access, any clinic could offer such capabilities, enabling global access to therapy personalization.

An AI-derived digital pathology-based biomarker to predict the benefit of androgen deprivation therapy in localized prostate cancer with validation in NRG/RTOG 9408.

223 Background: The current standard of care for men with intermediate- and high-risk localized prostate cancer treated with radiotherapy (RT) is the addition of androgen deprivation therapy (ADT). Presently, there are no validated predictive biomarkers to guide ADT use or duration in such men. Herein, we train and validate the first predictive biomarker for ADT use in prostate cancer using multiple phase III NRG Oncology randomized trials. Methods: Pre-treatment biopsy slides were digitized from five phase III NRG Oncology randomized trials of men receiving RT with or without ADT. The training set to develop the artificial intelligence (AI)-derived predictive biomarker included NRG/RTOG 9202, 9413, 9910, and 0126, and was trained to predict distant metastasis (DM). A multimodal deep learning architecture was developed to learn from both clinicopathologic and digital imaging histopathology data and identify differential outcomes by treatment type. After the model was locked, an independent biostatistician performed validation on NRG/RTOG 9408, a phase III randomized trial of RT +/- 4 months of ADT. The DM rates were calculated using cumulative incidence functions in biomarker positive and negative groups, and biomarker-treatment interaction was assessed using Fine-Gray regression such that death without DM was treated as a competing event. Results: Clinical and histopathological data was available for 5,654 of 7,957 eligible patients (71.1%). The training cohort included 3,935 patients and had a median follow-up of 13.6 years (IQR [10.2, 17.7]). After the AI-derived predictive ADT classifier was trained, it was validated in NRG/RTOG 9408 (n = 1719, median follow-up 17.6 years, IQR [15.0, 19.7]). In the NRG/RTOG 9408 validation cohort that had digital histopathology data, ADT significantly improved DM (HR 0.62, 95% CI [0.44, 0.87], p = 0.006), consistent with the published trial results. The biomarker-treatment interaction was significant (p-value = 0.0021). In patients with AI-biomarker positive disease (n = 673, 39%), ADT had a greater benefit compared to RT alone (HR 0.33, 95% CI [0.19, 0.57], p < 0.001). In the biomarker negative subgroup (n = 1046, 61%), the addition of ADT did not improve outcomes over RT alone (HR 1.00, 95% CI [0.64, 1.57], p = 0.99). The 15-year DM rate difference between RT versus RT+ADT in the biomarker negative group was 0.3%, vs biomarker positive group 9.4%. Conclusions: We have successfully validated in a phase III randomized trial the first predictive biomarker of ADT benefit with RT in localized intermediate risk prostate cancer using a novel AI-derived digital pathology-based platform. This AI-derived predictive biomarker demonstrates that a majority of patients treated with RT on NRG/RTOG 9408 did not require ADT and could have avoided the associated costs and side effects of this treatment.

Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data.

Hepatocellular carcinoma is the prevalent primary liver cancer, a silent disease that killed 782,000 worldwide in 2018. Multimodal deep learning is the application of deep learning techniques, fusing more than one data modality as the model's input. A computer-aided diagnosis system for hepatocellular carcinoma developed with multimodal deep learning approaches could use multiple data modalities as recommended by clinical guidelines, and enhance the robustness and the value of the second-opinion given to physicians. This article describes the process of creation and evaluation of an algorithm for computer-aided diagnosis of hepatocellular carcinoma developed with multimodal deep learning techniques fusing preprocessed computed-tomography images with structured data from patient Electronic Health Records. The classification performance achieved by the proposed algorithm in the test dataset was: accuracy = 86.9%, precision = 89.6%, recall = 86.9% and F-Score = 86.7%. These classification performance metrics are closer to the state-of-the-art in this area and were achieved with data modalities which are cheaper than traditional Magnetic Resonance Imaging approaches, enabling the use of the proposed algorithm by low and mid-sized healthcare institutions. The classification performance achieved with the multimodal deep learning algorithm is higher than human specialists diagnostic performance using only CT for diagnosis. Even though the results are promising, the multimodal deep learning architecture used for hepatocellular carcinoma prediction needs more training and test processes using different datasets before the use of the proposed algorithm by physicians in real healthcare routines. The additional training aims to confirm the classification performance achieved and enhance the model's robustness.

Leveraging Multimodal Deep Learning Architecture with Retina Lesion Information to Detect Diabetic Retinopathy.

PurposeTo improve disease severity classification from fundus images using a hybrid architecture with symptom awareness for diabetic retinopathy (DR).MethodsWe used 26,699 fundus images of 17,834 diabetic patients from three Taiwanese hospitals collected in 2007 to 2018 for DR severity classification. Thirty-seven ophthalmologists verified the images using lesion annotation and severity classification as the ground truth. Two deep learning fusion architectures were proposed: late fusion, which combines lesion and severity classification models in parallel using a postprocessing procedure, and two-stage early fusion, which combines lesion detection and classification models sequentially and mimics the decision-making process of ophthalmologists. Messidor-2 was used with 1748 images to evaluate and benchmark the performance of the architecture. The primary evaluation metrics were classification accuracy, weighted κ statistic, and area under the receiver operating characteristic curve (AUC).ResultsFor hospital data, a hybrid architecture achieved a good detection rate, with accuracy and weighted κ of 84.29% and 84.01%, respectively, for five-class DR grading. It also classified the images of early stage DR more accurately than conventional algorithms. The Messidor-2 model achieved an AUC of 97.09% in referral DR detection compared to AUC of 85% to 99% for state-of-the-art algorithms that learned from a larger database.ConclusionsOur hybrid architectures strengthened and extracted characteristics from DR images, while improving the performance of DR grading, thereby increasing the robustness and confidence of the architectures for general use.Translational RelevanceThe proposed fusion architectures can enable faster and more accurate diagnosis of various DR pathologies than that obtained in current manual clinical practice.

A Hybrid Method for Traffic Flow Forecasting Using Multimodal Deep Learning

Traffic flow forecasting has been regarded as a key problem of intelligent transport systems. In this work, we propose a hybrid multimodal deep learning method for short-term traffic flow forecasting, which can jointly and adaptively learn the spatial–temporal correlation features and long temporal interdependence of multi-modality traffic data by an attention auxiliary multimodal deep learning architecture. According to the highly nonlinear characteristics of multi-modality traffic data, the base module of our method consists of one-dimensional convolutional neural networks (1D CNN) and gated recurrent units (GRU) with the attention mechanism. The former is to capture the local trend features and the latter is to capture the long temporal dependencies. Then, we design a hybrid multimodal deep learning framework for fusing share representation features of different modality traffic data by multiple CNN-GRU-Attention modules. The experimental results indicate that the proposed multimodal deep learning model is capable of dealing with complex nonlinear urban traffic flow forecasting with satisfying accuracy and effectiveness.

Classification of pilots’ mental states using a multimodal deep learning network

An automation system for detecting the pilot's diversified mental states is an extremely important and essential technology, as it could prevent catastrophic accidents caused by the deteriorated cognitive state of pilots. Various types of biosignals have been employed to develop the system, since they accompany neurophysiological changes corresponding to the mental state transitions. In this study, we aimed to investigate the feasibility of a robust detection system of the pilot's mental states (i.e., distraction, workload, fatigue, and normal) based on multimodal biosignals (i.e., electroencephalogram, electrocardiogram, respiration, and electrodermal activity) and a multimodal deep learning (MDL) network. To do this, first, we constructed an experimental environment using a flight simulator in order to induce the different mental states and to collect the biosignals. Second, we designed the MDL architecture – which consists of a convolutional neural network and long short-term memory models – to efficiently combine the information of the different biosignals. Our experimental results successfully show that utilizing multimodal biosignals with the proposed MDL could significantly enhance the detection accuracy of the pilot's mental states.