Multimodal Deep Learning Model Research Articles

Assessing learners’ English public speaking anxiety with multimodal deep learning technologies

Public speaking anxiety (PSA) is a common phenomenon for language learners involving both psychological and physiological symptoms. Timely and effective PSA assessment can help diagnose learners’ speaking anxiety, offer learners feedback to alleviate their anxiety, and improve their public speaking competence. However, it is still a challenging issue to achieve accurate automated assessment of learners’ PSA due to the lack of large-scale and open-source multimodal datasets based on real classroom settings. This study collected the public speaking videos of English as a foreign language (EFL) learners in public speaking courses, and constructed a large-scale multimodal dataset named Speaking Anxiety in Real Classrooms (SARC) with three modalities of acoustic, visual, and textual data (including 1,158 manually-annotated speech videos and corresponding speech drafts of 382 participants). A multimodal deep learning model for automated assessment of learners’ speaking anxiety was proposed, and an online formative assessment platform was then developed to realize the automated assessment of PSA for classroom teaching. A pilot survey study involving 78 participants was then conducted to investigate learners’ acceptance of the platform. Experimental results verified the validity of the deep learning model and the consistency between automated assessment and teacher assessment. Learners’ acceptance data further indicated that collaboration between automated and human assessment provided them with the most satisfactory experience of using the platform to improve their English public speaking.

Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning.

The objective of this study was to provide a multi-modal deep learning framework for forecasting the survival of rectal cancer patients by utilizing both digital pathological images data and non-imaging clinical data. The research included patients diagnosed with rectal cancer by pathological confirmation from January 2015 to December 2016. Patients were allocated to training and testing sets in a randomized manner, with a ratio of 4:1. The tissue microarrays (TMAs) and clinical indicators were obtained. Subsequently, we selected distinct deep learning models to individually forecast patient survival. We conducted a scanning procedure on the TMAs in order to transform them into digital pathology pictures. Additionally, we performed pre-processing on the clinical data of the patients. Subsequently, we selected distinct deep learning algorithms to conduct survival prediction analysis using patients' pathological images and clinical data, respectively. A total of 292 patients with rectal cancer were randomly allocated into two groups: a training set consisting of 234 cases, and a testing set consisting of 58 instances. Initially, we make direct predictions about the survival status by using pre-processed Hematoxylin and Eosin (H&E) pathological images of rectal cancer. We utilized the ResNest model to extract data from histopathological images of patients, resulting in a survival status prediction with an AUC (Area Under the Curve) of 0.797. Furthermore, we employ a multi-head attention fusion (MHAF) model to combine image features and clinical features in order to accurately forecast the survival rate of rectal cancer patients. The findings of our experiment show that the multi-modal structure works better than directly predicting from histopathological images. It achieves an AUC of 0.837 in predicting overall survival (OS). Our study highlights the potential of multi-modal deep learning models in predicting survival status from histopathological images and clinical information, thus offering valuable insights for clinical applications.

Predicting rectal cancer tumor budding grading based on MRI and CT with multimodal deep transfer learning: A dual-center study

ObjectiveTo investigate the effectiveness of a multimodal deep learning model in predicting tumor budding (TB) grading in rectal cancer (RC) patients. Materials and methodsA retrospective analysis was conducted on 355 patients with rectal adenocarcinoma from two different hospitals. Among them, 289 patients from our institution were randomly divided into an internal training cohort (n = 202) and an internal validation cohort (n = 87) in a 7:3 ratio, while an additional 66 patients from another hospital constituted an external validation cohort. Various deep learning models were constructed and compared for their performance using T1CE and CT-enhanced images, and the optimal models were selected for the creation of a multimodal fusion model. Based on single and multiple factor logistic regression, clinical N staging and fecal occult blood were identified as independent risk factors and used to construct the clinical model. A decision-level fusion was employed to integrate these two models to create an ensemble model. The predictive performance of each model was evaluated using the area under the curve (AUC), DeLong's test, calibration curve, and decision curve analysis (DCA). Model visualization Gradient-weighted Class Activation Mapping (Grad-CAM) was performed for model interpretation. ResultsThe multimodal fusion model demonstrated superior performance compared to single-modal models, with AUC values of 0.869 (95% CI: 0.761–0.976) for the internal validation cohort and 0.848 (95% CI: 0.721–0.975) for the external validation cohort. N-stage and fecal occult blood were identified as clinically independent risk factors through single and multivariable logistic regression analysis. The final ensemble model exhibited the best performance, with AUC values of 0.898 (95% CI: 0.820–0.975) for the internal validation cohort and 0.868 (95% CI: 0.768–0.968) for the external validation cohort. ConclusionMultimodal deep learning models can effectively and non-invasively provide individualized predictions for TB grading in RC patients, offering valuable guidance for treatment selection and prognosis assessment.

Integrating wearable sensor data and self-reported diaries for personalized affect forecasting

Emotional states, as indicators of affect, are pivotal to overall health, making their accurate prediction before onset crucial. Current studies are primarily centered on immediate short-term affect detection using data from wearable and mobile devices. These studies typically focus on objective sensory measures, often neglecting other forms of self-reported information like diaries and notes. In this paper, we propose a multimodal deep learning model for affect status forecasting. This model combines a transformer encoder with a pre-trained language model, facilitating the integrated analysis of objective metrics and self-reported diaries. To validate our model, we conduct a longitudinal study, enrolling college students and monitoring them over a year, to collect an extensive dataset including physiological, environmental, sleep, metabolic, and physical activity parameters, alongside open-ended textual diaries provided by the participants. Our results demonstrate that the proposed model achieves predictive accuracy of 82.50% for positive affect and 82.76% for negative affect, a full week in advance. The effectiveness of our model is further elevated by its explainability.

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Abstract Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2313.

EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN.

Analysis of brain signals is essential to the study of mental states and various neurological conditions. The two most prevalent noninvasive signals for measuring brain activities are electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, characterized by its higher sampling frequency, captures more temporal features, while fNIRS, with a greater number of channels, provides richer spatial information. Although a few previous studies have explored the use of multimodal deep-learning models to analyze brain activity for both EEG and fNIRS, subject-independent training-testing split analysis remains underexplored. The results of the subject-independent setting directly show the model's ability on unseen subjects, which is crucial for real-world applications. In this paper, we introduce EF-Net, a new CNN-based multimodal deep-learning model. We evaluate EF-Net on an EEG-fNIRS word generation (WG) dataset on the mental state recognition task, primarily focusing on the subject-independent setting. For completeness, we report results in the subject-dependent and subject-semidependent settings as well. We compare our model with five baseline approaches, including three traditional machine learning methods and two deep learning methods. EF-Net demonstrates superior performance in both accuracy and F1 score, surpassing these baselines. Our model achieves F1 scores of 99.36%, 98.31%, and 65.05% in the subject-dependent, subject-semidependent, and subject-independent settings, respectively, surpassing the best baseline F1 scores by 1.83%, 4.34%, and 2.13% These results highlight EF-Net's capability to effectively learn and interpret mental states and brain activity across different and unseen subjects.

Predicting Obstructive Sleep Apnea Based on Computed Tomography Scan Using Deep Learning Models.

The incidence of clinically undiagnosed obstructive sleep apnea (OSA) is high among the general population due to limited access to polysomnography. Computed tomography (CT) of craniofacial regions obtained for other purposes can be beneficial in predicting OSA and its severity. To predict OSA and its severity based on paranasal CT using a 3-dimensional deep learning algorithm. One internal dataset (n=798) and two external datasets (n=135 and 85) were used in this study. In the internal dataset, 92 normal, 159 mild, 201 moderate, and 346 severe OSA participants were enrolled to derive the deep learning model. A multimodal deep learning model was elicited from the connection between a 3-dimensional convolutional neural network (CNN)-based part treating unstructured data (CT images) and a multi-layer perceptron (MLP)-based part treating structured data (age, sex, and body mass index) to predict OSA and its severity. In four-class classification for predicting the severity of OSA, the AirwayNet-MM-H model (multimodal model with airway-highlighting preprocessing algorithm) showed an average accuracy of 87.6% (95% confidence interval [CI] 86.8-88.6) in the internal dataset and 84.0% (95% CI 83.0-85.1) and 86.3% (95% CI 85.3-87.3) in the two external datasets, respectively. In the two-class classification for predicting significant OSA (moderate to severe OSA), The area under the receiver operating characteristics (AUROC), accuracy, sensitivity, specificity, and F1 score were 0.910 (95% CI 0.899-0.922), 91.0% (95% CI 90.1-91.9), 89.9% (95% CI 88.8-90.9), 93.5% (95% CI 92.7-94.3), and 93.2% (95% CI 92.5-93.9), respectively, in the internal dataset. Furthermore, the diagnostic performance of the Airway Net-MM-H model outperformed that of the other six state-of-the-art deep learning models in terms of accuracy for both four- and two-class classifications and AUROC for two-class classification (p<0.001). A novel deep learning model, including a multimodal deep learning model and an airway-highlighting preprocessing algorithm from CT images obtained for other purposes, can provide significantly precise outcomes for OSA diagnosis.

Predicting long-term progression of Alzheimer’s disease using a multimodal deep learning model incorporating interaction effects

BackgroundIdentifying individuals with mild cognitive impairment (MCI) at risk of progressing to Alzheimer’s disease (AD) provides a unique opportunity for early interventions. Therefore, accurate and long-term prediction of the conversion from MCI to AD is desired but, to date, remains challenging. Here, we developed an interpretable deep learning model featuring a novel design that incorporates interaction effects and multimodality to improve the prediction accuracy and horizon for MCI-to-AD progression.MethodsThis multi-center, multi-cohort retrospective study collected structural magnetic resonance imaging (sMRI), clinical assessments, and genetic polymorphism data of 252 patients with MCI at baseline from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. Our deep learning model was cross-validated on the ADNI-1 and ADNI-2/GO cohorts and further generalized in the ongoing ADNI-3 cohort. We evaluated the model performance using the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, and F1 score.ResultsOn the cross-validation set, our model achieved superior results for predicting MCI conversion within 4 years (AUC, 0.962; accuracy, 92.92%; sensitivity, 88.89%; specificity, 95.33%) compared to all existing studies. In the independent test, our model exhibited consistent performance with an AUC of 0.939 and an accuracy of 92.86%. Integrating interaction effects and multimodal data into the model significantly increased prediction accuracy by 4.76% (P = 0.01) and 4.29% (P = 0.03), respectively. Furthermore, our model demonstrated robustness to inter-center and inter-scanner variability, while generating interpretable predictions by quantifying the contribution of multimodal biomarkers.ConclusionsThe proposed deep learning model presents a novel perspective by combining interaction effects and multimodality, leading to more accurate and longer-term predictions of AD progression, which promises to improve pre-dementia patient care.

Multi-modality deep learning model reaches high prediction accuracy in the diagnosis of ovarian cancer

We evaluated the diagnostic performance of a multimodal deep-learning (DL) model for ovarian mass differential diagnosis. This single-center retrospective study included 1,054 ultrasound (US)-detected ovarian tumors (699 benign and 355 malignant). Patients were randomly divided into training (n= 675), validation (n= 169), and testing (n= 210) sets. The model was developed using ResNet-50. Three DL-based models were proposed for benign-malignant classification of these lesions: single-modality model that only utilized US images; dual-modality model that used US images and menopausal status as inputs; and multi-modality model that integrated US images, menopausal status, and serum indicators. After 5-fold cross-validation, 210 lesions were tested. We evaluated the three models using the area under the curve (AUC), accuracy, sensitivity, and specificity. The multimodal model outperformed the single- and dual-modality models with 93.80% accuracy and 0.983 AUC. The Multimodal ResNet-50 DL model outperformed the single- and dual-modality models in identifying benign and malignant ovarian tumors.

Predicting microvascular invasion in hepatocellular carcinoma with a CT- and MRI-based multimodal deep learning model.

To investigate the value of a multimodal deep learning (MDL) model based on computed tomography (CT) and magnetic resonance imaging (MRI) for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC). A total of 287 patients with HCC from our institution and 58 patients from another individual institution were included. Among these, 119 patients with only CT data and 116 patients with only MRI data were selected for single-modality deep learning model development, after which select parameters were migrated for MDL model development with transfer learning (TL). In addition, 110 patients with simultaneous CT and MRI data were divided into a training cohort (n = 66) and a validation cohort (n = 44). We input the features extracted from DenseNet121 into an extreme learning machine (ELM) classifier to construct a classification model. The area under the curve (AUC) of the MDL model was 0.844, which was superior to that of the single-phase CT (AUC = 0.706-0.776, P < 0.05), single-sequence MRI (AUC = 0.706-0.717, P < 0.05), single-modality DL model (AUCall-phase CT = 0.722, AUCall-sequence MRI = 0.731; P < 0.05), clinical (AUC = 0.648, P < 0.05), but not to that of the delay phase (DP) and in-phase (IP) MRI and portal venous phase (PVP) CT models. The MDL model achieved better performance than models described above (P < 0.05). When combined with clinical features, the AUC of the MDL model increased from 0.844 to 0.871. A nomogram, combining deep learning signatures (DLS) and clinical indicators for MDL models, demonstrated a greater overall net gain than the MDL models (P < 0.05). The MDL model is a valuable noninvasive technique for preoperatively predicting MVI in HCC.

Analysing the spatiotemporal variation and influencing factors of Lake Chaohu's CDOM over the past 40 years using machine learning

AbstractChromophoric dissolved organic matter (CDOM) in aquatic environments is an important component of the biogeochemical cycle and carbon cycle. The aim of this study is to investigate the long‐term changes in CDOM in shallow and eutrophic Chaohu Lake, as well as its relationship with climate, environment and social factors. Using long time series Landsat image data and machine learning technology, the spatiotemporal evolution of Chaohu CDOM since 1987 was reconstructed. A total of 180 samples were collected, which were divided into three parts based on regional and hydrological characteristics. The results show that the water quality in different regions were significantly different, and TN may be the key factor driving the change of CDOM in Chaohu Lake. Machine learning algorithms including random forest (RF), support vector regression (SVR), neural network (NN), multimodal deep learning (MDL) model and Extreme Gradient Boosting (XGBoost) were used, among which XGBoost model performed best (R2 = 0.955, mean absolute error [MAE] = 0.024 mg/L, root mean square error [RMSE] = 0.036 mg/L, bias = 1.005) and was used for CDOM spatiotemporal variation retrieval. The change of CDOM was seasonal, highest in August (0.67 m−1) and lowest in December (0.48 m−1), and the western lake is the main source of CDOM. Annual variability of the CDOM indicates that it began to decline after the completion of water pollution control in 2000. Temperature changes were closely related to CDOM (P &lt; 0.01) and agricultural non‐point source pollution plays an important role in Chaohu Lake. This study will provide feasible methods and scientific basis for the long‐term remote sensing supervision of CDOM.

Multimodal Deep Learning for Estimating Lane-Level Urban Traffic by Fusing Closed-Circuit Television and Dedicated Short-Range Communication Data

Seamless, accurate, and reliable traffic information is critical for planning and operational strategies in traffic management. Urban networks are increasingly being equipped with various types of detectors to collect continuous traffic data. However, the widespread installation and management of these detectors at all intersections and road sections are limited by cost constraints and various problems. Moreover, the direct integration and utilization of heterogeneous data sources are challenging because of their distinct and interconnected characteristics. This study introduces a multimodal deep learning model, combining closed-circuit television (CCTV) and dedicated short-range communication (DSRC) data, to estimate lane-level urban traffic volumes. The proposed model employs a multilayer perceptron to extract features from each modality. These features are then fused and used as input into a recurrent neural network model that estimates the lane-level traffic volume. We present the multimodal deep learning model in three forms: (1) fusion of traffic volume, occupancy, and queue length from CCTV data; (2) fusion of traffic volume from CCTV data with travel time from DSRC data; and (3) fusion of different attributes from CCTV and heterogeneous DSRC data. In addition, we develop a single-modality model that solely utilizes CCTV data on traffic volume to compare the performance with the proposed multimodal model and identify scenarios where the multimodal approach is essential. The proposed model demonstrates significant improvements over the single-modality model, providing enhanced accuracy for higher temporal resolutions and lanes that permit left or right turns.

Predicting clinical outcomes in the S1314-COXEN trial using a multimodal deep learning model integrating histopathology, cell types, and gene expression.

533 Background: Accurate prediction of response to neoadjuvant chemotherapy (NAC) is essential for optimizing treatment outcomes in muscle-invasive bladder cancer (MIBC). We present a cutting-edge multimodal deep learning model aimed at predicting outcomes of NAC from accessible H &amp; E images and molecular data. Methods: We designed a Graph-based Multi-modal Late Fusion (GMLF) deep learning model that integrates three types of data from the S1314-COXEN clinical trial: 1) Neural embeddings from 182 whole slide images (WSIs), generated via the ResNet50 architecture. 2) Cell types based on morphology, including cancer, necrotic, immune, and stromal cells, and their spatial positions within the tumor microenvironment extracted from WSIs using the HoVer-Net framework, a multiple-branch convolutional neural network for predicting cell types. 3) Patient-level RNA expression data derived from 1,071-dimensional gene expression vectors. The dataset was randomly divided into 80% for model training, using 5-fold cross-validation (5-fold CV), and 20% for internal validation. Results: Our dataset included 182 WSIs of 180 patients who were randomized to receive either gemcitabine-cisplatin or dose-dense methotrexate-vinblastine-adriamycin-doxorubicin-cisplatin. Of all patients, 30.8% demonstrated a complete pathological response (pT0 after radical cystectomy). Our GMLF model achieved an AUROC of 0.7417 ± 0.1021 for predicting whether a patient showed a complete pathologic response to NAC in a 5-fold CV. Using an 80/20 training and validation split, the model yielded an AUROC of 0.7236. Interestingly, our model emphasized the contribution of spatial information collected from WSIs. For selecting the best architecture for the histology branch, the Slidegraph+ framework, a graph neural network-based model, achieved an AUROC of 0.6938 ± 0.0565, significantly outperforming patch-level models as the second-best, which achieved AUROC of 0.5572 ± 0.1793. Shapley Additive Explanation (SHAP) analysis revealed RNA expression branch as the most influential, with a mean SHAP magnitude of 0.13, followed by the neural embeddings at 0.12. SHAP-based interpretation identified the most important expressed genes impacting model predictability, including TP63. Our gene set analysis identified the most influential pathways for predicting response, which included basal differentiation and myo-fibroblast enrichment (p-value &lt; 0.05). Conclusions: Our interpretable GMLF model accurately predicts NAC response in patients with MIBC by integrating standard H&amp;E images and RNA expression data. Our model interpretations not only revealed the importance of each modality but also uncovered histopathological, cellular, and molecular underpinnings of response to NAC in MIBC. These findings open the door to AI-guided development of precision therapies for MIBC. Clinical trial information: NCT02177695 .

A Multi-Modal Deep-Learning Air Quality Prediction Method Based on Multi-Station Time-Series Data and Remote-Sensing Images: Case Study of Beijing and Tianjin.

The profound impacts of severe air pollution on human health, ecological balance, and economic stability are undeniable. Precise air quality forecasting stands as a crucial necessity, enabling governmental bodies and vulnerable communities to proactively take essential measures to reduce exposure to detrimental pollutants. Previous research has primarily focused on predicting air quality using only time-series data. However, the importance of remote-sensing image data has received limited attention. This paper proposes a new multi-modal deep-learning model, Res-GCN, which integrates high spatial resolution remote-sensing images and time-series air quality data from multiple stations to forecast future air quality. Res-GCN employs two deep-learning networks, one utilizing the residual network to extract hidden visual information from remote-sensing images, and another using a dynamic spatio-temporal graph convolution network to capture spatio-temporal information from time-series data. By extracting features from two different modalities, improved predictive performance can be achieved. To demonstrate the effectiveness of the proposed model, experiments were conducted on two real-world datasets. The results show that the Res-GCN model effectively extracts multi-modal features, significantly enhancing the accuracy of multi-step predictions. Compared to the best-performing baseline model, the multi-step prediction's mean absolute error, root mean square error, and mean absolute percentage error increased by approximately 6%, 7%, and 7%, respectively.

Early identification of stroke through deep learning with multi-modal human speech and movement data

JOURNAL/nrgr/04.03/01300535-202501000-00031/figure1/v/2024-05-10T114357Z/r/image-tiff Early identification and treatment of stroke can greatly improve patient outcomes and quality of life. Although clinical tests such as the Cincinnati Pre-hospital Stroke Scale (CPSS) and the Face Arm Speech Test (FAST) are commonly used for stroke screening, accurate administration is dependent on specialized training. In this study, we proposed a novel multimodal deep learning approach, based on the FAST, for assessing suspected stroke patients exhibiting symptoms such as limb weakness, facial paresis, and speech disorders in acute settings. We collected a dataset comprising videos and audio recordings of emergency room patients performing designated limb movements, facial expressions, and speech tests based on the FAST. We compared the constructed deep learning model, which was designed to process multi-modal datasets, with six prior models that achieved good action classification performance, including the I3D, SlowFast, X3D, TPN, TimeSformer, and MViT. We found that the findings of our deep learning model had a higher clinical value compared with the other approaches. Moreover, the multi-modal model outperformed its single-module variants, highlighting the benefit of utilizing multiple types of patient data, such as action videos and speech audio. These results indicate that a multi-modal deep learning model combined with the FAST could greatly improve the accuracy and sensitivity of early stroke identification of stroke, thus providing a practical and powerful tool for assessing stroke patients in an emergency clinical setting.

Multimodal deep learning model for Covid-19 detection

This study introduces a novel multimodal deep learning model for detecting Covid-19, uniquely combining chest X-ray images and cough sound features. We incorporated an attention layer to effectively influence the inputs from both features to enhance the detection process. The integration of time and frequency domain cough features with chest X-ray images represents a significant advancement in Covid-19 detection methodologies, offering promising results and potential applications in the medical field. In the proposed model, we modified vgg16 and a faster region-based convolutional neural network (FRCNN). The modified vgg16 is used to extract the features from both datasets, whereas the modified FRCNN is used for detection. Utilizing the Vgg16 model for feature extraction and faster-rcnn for Covid-19 detection allows for a robust and efficient approach. We analyzed the performance of the proposed methodology using a dataset of 2059 Covid-19 positive chest X-ray images and a diverse collection of 25,000 cough recordings, including 716 positive, 7693 negative, and 1591 symptomatic cases. The datasets were divided into 80% for training and 20% for testing. The model achieves 99.80% accuracy, an f1-score of 99.70%, and 0.0132 validation loss. We further performed an extensive ablation experiment on the features. We compared the results of the proposed model to already existing networks using chest X-rays and coughs as datasets, and the proposed model outperformed them. Experiments also show that the trained model resists training data quantity and quality and can effectively detect Covid-19 in chest X-rays and coughs.

Prediction of osteoporosis using MRI and CT scans with unimodal and multimodal deep-learning models.

Osteoporosis is the systematic degeneration of the human skeleton, with consequences ranging from a reduced quality of life to mortality. Therefore, the prediction of osteoporosis reduces risks and supports patients in taking precautions. Deep-learning and specific models achieve highly accurate results using different imaging modalities. The primary purpose of this research was to develop unimodal and multimodal deep-learning-based diagnostic models to predict bone mineral loss of the lumbar vertebrae using magnetic resonance (MR) and computed tomography (CT) imaging. Patients who received both lumbar dual-energy X-ray absorptiometry (DEXA) and MRI (n = 120) or CT (n = 100) examinations were included in this study. Unimodal and multimodal convolutional neural networks (CNNs) with dual blocks were proposed to predict osteoporosis using lumbar vertebrae MR and CT examinations in separate and combined datasets. Bone mineral density values obtained by DEXA were used as reference data. The proposed models were compared with a CNN model and six benchmark pre-trained deep-learning models. The proposed unimodal model obtained 96.54%, 98.84%, and 96.76% balanced accuracy for MRI, CT, and combined datasets, respectively, while the multimodal model achieved 98.90% balanced accuracy in 5-fold cross-validation experiments. Furthermore, the models obtained 95.68%-97.91% accuracy with a hold-out validation dataset. In addition, comparative experiments demonstrated that the proposed models yielded superior results by providing more effective feature extraction in dual blocks to predict osteoporosis. This study demonstrated that osteoporosis was accurately predicted by the proposed models using both MR and CT images, and a multimodal approach improved the prediction of osteoporosis. With further research involving prospective studies with a larger number of patients, there may be an opportunity to implement these technologies into clinical practice.

A Study of Mainstream Media Communication and Audience Participatory Culture Based on Multimodal Analysis Approach

Abstract This paper analyzes the network expression and audience participatory discourse system under the mainstream media communication mode, combines the audiovisual multimodal audience cognitive participation and the multimodal features of the mainstream media, and proposes a multimodal deep learning model based on mainstream media communication. Based on the picture vision and human emotion migration methods, the picture vision feature migration module and the text emotion migration module are constructed respectively, and the multimodal feature fusion model is built by applying the gradient enhancement decision tree. Linguistic modality analysis of mainstream media communication discourse is carried out by analyzing the development characteristics of audience participatory cultural communication modes. The influence of different mainstream media communication discourses on the audience’s participatory culture is examined through logistic regression analysis while screening network discourse access. The regression analysis shows that the mainstream media communication modes represented by social software and microblogging have a significant impact on the development of the audience’s participatory culture, with p-values of 0.0452 and 0.0068, respectively, and thus the mainstream media need to strengthen content innovation, focus on specific audience groups, and make full use of new media platforms to provide audiences with richer, more accurate and professional news services.

An emotion-driven and topic-aware dialogue framework for human–robot interaction

ABSTRACT Developing dialogue services for robots has been promoted nowadays for providing natural human–robot interactions to enhance user experiences as conversation is a key instrument for creating and maintaining their mutual relationships. In this work, we present a trainable framework for modeling context-aware human–robot dialogues, and apply it to a live streaming application to demonstrate the need for social robots. Our social chatting robot framework takes into account both emotional context and topic-focused content information to generate appropriate responses. This framework has some unique features. It adopts a multimodal deep learning model to recognize the participants’ emotions. Moreover, our work includes a topic-aware neural model that enables the social robot to follow certain conversation topics to fulfill the specific dialoguing goals and to enhance the chatting coherence. Most importantly, we design a strategy able to take emotions as the drive to control over the neural method of generating utterances. To evaluate the proposed approach, we have conducted several sets of quantitative and quantitative experiments. The results highlight the importance of multimodal emotion recognition and topic-awareness in dialoguing and confirm the feasibility and effectiveness of our framework for human–robot interaction.

Unveiling pedestrian injury risk factors through integration of urban contexts using multimodal deep learning

This study aimed to identify contributing risk factors for pedestrian injury by integrating socio-spatial and street-level contexts through multimodal deep learning to overcome the limitations of existing studies that only consider one type of data. To investigate how the two contexts assist in describing pedestrian injury risk, six multimodal deep learning models were established by varying the ratio integrating the two contexts. The developed model with the highest performance was interpreted by using two XAI methods: SHAP for socio-spatial context and Grad-CAM for street-level context. The results indicated that the street-level context mainly contributes to the pedestrian injury risk level, assisted by the socio-spatial context, which cannot be captured at the street-level. The three main contributing risk factors were identified through model interpretation: the fragmented sky view due to the locations of high-rise buildings, the placement of crosswalks in areas adjacent to public transits, and interregional sociodemographic disparities. This study provides insight into the use of integrating two different urban contexts to identify pedestrian injury risk factors, which are expected to support improvement strategies that enhance public health.