Multimodal Deep Learning Framework Research Articles

A dynamic multi-modal deep reinforcement learning framework for 3D bin packing problem

The 3D bin packing problem, a notorious NP-hard combinatorial optimization challenge with wide-ranging practical implications, focuses on optimizing spatial allocation within a bin through the arrangement of boxes. Previous studies have shown promise in employing Neural Combinatorial Optimization to tackle such intractable problems. However, due to the inherent diversity in observations and the sparse rewards, current learning-based methods for addressing the 3D bin packing problem have yielded less-than-ideal outcomes. In response to this shortfall, we propose a novel Dynamic Multi-modal deep Reinforcement Learning framework tailored specifically for the 3D Bin Packing Problem, coined as DMRL-BPP. This framework stands apart from existing learning-based bin-packing approaches in two pivotal respects. Firstly, in order to capture the range of observations, we introduce an innovative dynamic multi-modal encoder, comprising dynamic boxes state and gradient height map sub-encoders, effectively modeling multi-modal information. Secondly, to mitigate the challenge of sparse rewards, we put forth a novel reward function, offering an efficient and adaptable approach for addressing the 3D bin packing problem. Extensive experimental results validate the superiority of our method over all baseline approaches, across instances of varying scales.

MMSyn: A New Multimodal Deep Learning Framework for Enhanced Prediction of Synergistic Drug Combinations.

Combination therapy is a promising strategy for the successful treatment of cancer. The large number of possible combinations, however, mean that it is laborious and expensive to screen for synergistic drug combinations in vitro. Nevertheless, because of the availability of high-throughput screening data and advances in computational techniques, deep learning (DL) can be a useful tool for the prediction of synergistic drug combinations. In this study, we proposed a multimodal DL framework, MMSyn, for the prediction of synergistic drug combinations. First, features embedded in the drug molecules were extracted: structure, fingerprint, and string encoding. Then, gene expression data, DNA copy number, and pathway activity were used to describe cancer cell lines. Finally, these processed features were integrated using an attention mechanism and an interaction module and then input into a multilayer perceptron to predict drug synergy. Experimental results showed that our method outperformed five state-of-the-art DL methods and three traditional machine learning models for drug combination prediction. We verified that MMSyn achieved superior performance in stratified cross-validation settings using both the drug combination and cell line data. Moreover, we performed a set of ablation experiments to illustrate the effectiveness of each component and the efficacy of our model. In addition, our visual representation and case studies further confirmed the effectiveness of our model. All results showed that MMSyn can be used as a powerful tool for the prediction of synergistic drug combinations.

Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning.

The objective of this study was to provide a multi-modal deep learning framework for forecasting the survival of rectal cancer patients by utilizing both digital pathological images data and non-imaging clinical data. The research included patients diagnosed with rectal cancer by pathological confirmation from January 2015 to December 2016. Patients were allocated to training and testing sets in a randomized manner, with a ratio of 4:1. The tissue microarrays (TMAs) and clinical indicators were obtained. Subsequently, we selected distinct deep learning models to individually forecast patient survival. We conducted a scanning procedure on the TMAs in order to transform them into digital pathology pictures. Additionally, we performed pre-processing on the clinical data of the patients. Subsequently, we selected distinct deep learning algorithms to conduct survival prediction analysis using patients' pathological images and clinical data, respectively. A total of 292 patients with rectal cancer were randomly allocated into two groups: a training set consisting of 234 cases, and a testing set consisting of 58 instances. Initially, we make direct predictions about the survival status by using pre-processed Hematoxylin and Eosin (H&E) pathological images of rectal cancer. We utilized the ResNest model to extract data from histopathological images of patients, resulting in a survival status prediction with an AUC (Area Under the Curve) of 0.797. Furthermore, we employ a multi-head attention fusion (MHAF) model to combine image features and clinical features in order to accurately forecast the survival rate of rectal cancer patients. The findings of our experiment show that the multi-modal structure works better than directly predicting from histopathological images. It achieves an AUC of 0.837 in predicting overall survival (OS). Our study highlights the potential of multi-modal deep learning models in predicting survival status from histopathological images and clinical information, thus offering valuable insights for clinical applications.

An Integrated Multimodal Deep Learning Framework for Accurate Skin Disease Classification

In order to effectively treat skin diseases, an accurate and prompt diagnosis is required. In this article, a novel method for classifying skin disorders using a multimodal classifier is presented. The proposed classifier utilizes multiple information sources to enhance the accuracy of disease classification. It incorporates images of skin lesions and patient-specific data. The multimodal classifier simultaneously classifies diseases by combining image and structured data inputs. The effectiveness of the proposed classifier was evaluated using the ISIC 2018 dataset, which includes images and clinical data for seven categories of skin diseases. The results indicate that the proposed model outperforms conventional single-modal and single-task classifiers, achieving an accuracy of 98.66% for image classification and 94.40% for clinical data classification. In addition, we compare the performance of the proposed model with that of other methodologies, demonstrating its superiority. Despite yielding promising results, the proposed method has limitations in terms of data requirements and generalizability. Future research directions include incorporating additional information sources, investigating genetic data integration, and applying the method to various medical conditions. This study illustrates the potential of integrating multimodal techniques with transfer learning in deep neural networks to enhance the classification accuracy of cutaneous diseases.

Language-assisted deep learning for autistic behaviors recognition

Correctly recognizing the behaviors of children with Autism Spectrum Disorder (ASD) is of vital importance for the diagnosis of Autism and timely early intervention. However, the observation and recording during the treatment from the parents of autistic children may not be accurate and objective. In such cases, automatic recognition systems based on computer vision and machine learning (in particular deep learning) technology can alleviate this issue to a large extent. Existing human action recognition models can now achieve impressive performance on challenging activity datasets, e.g., daily activity, and sports activity. However, problem behaviors in children with ASD are very different from these general activities, and recognizing these problem behaviors via computer vision is less studied. In this paper, we first evaluate a strong baseline for action recognition, i.e., Video Swin Transformer, on two autism behaviors datasets (SSBD and ESBD) and show that it can achieve high accuracy and outperform the previous methods by a large margin, demonstrating the feasibility of vision-based problem behaviors recognition. Moreover, we propose language-assisted training to further enhance the action recognition performance. Specifically, we develop a two-branch multimodal deep learning framework by incorporating the ”freely available” language description for each type of problem behavior. Experimental results demonstrate that incorporating additional language supervision can bring an obvious performance boost for the autism problem behaviors recognition task as compared to using the video information only (i.e., 3.49% improvement on ESBD and 1.46% on SSBD). Our code and model will be publicly available for reproducing the results.

A Multimodal Deep Learning Framework for Predicting PPI-Modulator Interactions.

Protein-protein interactions (PPIs) are essential for various biological processes and diseases. However, most existing computational methods for identifying PPI modulators require either target structure or reference modulators, which restricts their applicability to novel PPI targets. To address this challenge, we propose MultiPPIMI, a sequence-based deep learning framework that predicts the interaction between any given PPI target and modulator. MultiPPIMI integrates multimodal representations of PPI targets and modulators and uses a bilinear attention network to capture intermolecular interactions. Experimental results on our curated benchmark data set show that MultiPPIMI achieves an average AUROC of 0.837 in three cold-start scenarios and an AUROC of 0.994 in the random-split scenario. Furthermore, the case study shows that MultiPPIMI can assist molecular docking simulations in screening inhibitors of Keap1/Nrf2 PPI interactions. We believe that the proposed method provides a promising way to screen PPI-targeted modulators.

SinkholeNet: A novel RGB-slope sinkhole dataset and deep weakly-supervised learning framework for sinkhole classification and localization

This paper proposes a novel multimodal deep weakly-supervised learning framework, SinkholeNet, to classify and localize sinkhole(s) in high-resolution RGB-slope aerial images. The SinkholeNet first employs a multimodal Convolutional Neural Network (CNN) architecture that simultaneously extracts features from the input RGB image and ground slope map and then fuses the extracted features. It then uses an improved ShuffleNet architecture on the fused features to classify patches as sinkholes or non-sinkholes. Finally, the last extracted feature maps, belonging to the sinkhole class, are used as input of gradient-weighted class activation mapping (Grad-CAM) to localize sinkhole(s) in a weakly-supervised setting. The proposed weakly-supervised framework intends to increase the available labeled data for training and decrease the cost of human annotation. We also introduce a novel publicly available weakly labeled sinkhole dataset comprising RGB-slope paired image patches to support reproducible research. The experimental results on the newly introduced dataset show that the SinkholeNet outperforms the other methods considered in this paper both for sinkhole classification and localization.

Predicting multiple sclerosis severity with multimodal deep neural networks

Multiple Sclerosis (MS) is a chronic disease developed in the human brain and spinal cord, which can cause permanent damage or deterioration of the nerves. The severity of MS disease is monitored by the Expanded Disability Status Scale, composed of several functional sub-scores. Early and accurate classification of MS disease severity is critical for slowing down or preventing disease progression via applying early therapeutic intervention strategies. Recent advances in deep learning and the wide use of Electronic Health Records (EHR) create opportunities to apply data-driven and predictive modeling tools for this goal. Previous studies focusing on using single-modal machine learning and deep learning algorithms were limited in terms of prediction accuracy due to data insufficiency or model simplicity. In this paper, we proposed the idea of using patients’ multimodal longitudinal and longitudinal EHR data to predict multiple sclerosis disease severity in the future. Our contribution has two main facets. First, we describe a pioneering effort to integrate structured EHR data, neuroimaging data and clinical notes to build a multi-modal deep learning framework to predict patient’s MS severity. The proposed pipeline demonstrates up to 19% increase in terms of the area under the Area Under the Receiver Operating Characteristic curve (AUROC) compared to models using single-modal data. Second, the study also provides valuable insights regarding the amount useful signal embedded in each data modality with respect to MS disease prediction, which may improve data collection processes.

Multimodal Deep Learning Framework for Book Recommendations: Harnessing Image Processing with VGG16 and Textual Analysis via LSTM-Enhanced Word2Vec

Multimodal Deep Learning Framework for Book Recommendations: Harnessing Image Processing with VGG16 and Textual Analysis via LSTM-Enhanced Word2Vec

A Comprehensive and Versatile Multimodal Deep-Learning Approach for Predicting Diverse Properties of Advanced Materials.

A multimodal deep-learning (MDL) framework is presented for predicting physical properties of a ten-dimensional acrylic polymer composite material by merging physical attributes and chemical data. The MDL model comprises four modules, including three generative deep-learning models for material structure characterization and a fourth model for property prediction. The approach handles an 18-dimensional complexity, with ten compositional inputs and eight property outputs, successfully predicting 913 680 property data points across 114 210 composition conditions. This level of complexity is unprecedented in computational materials science, particularly for materials with undefined structures. A framework is proposed to analyze the high-dimensional information space for inverse material design, demonstrating flexibility and adaptability to various materials and scales, provided sufficient data are available. This study advances future research on different materials and the development of more sophisticated models, drawing the authors closer to the ultimate goal of predicting all properties of all materials.

A Novel Multimodal Deep Learning Framework for Encrypted Traffic Classification

Traffic classification is essential for cybersecurity maintenance and network management, and has been widely used in QoS (Quality of Service) guarantees, intrusion detection, and other tasks. Recently, with the emergence of SSL/TLS encryption protocols in the modern Internet environment, the traditional payload-based classification methods are no longer effective. Some researchers have used machine learning methods to model the flow features of encrypted traffics (e.g. message type, length sequence, statistical features, etc.), and achieved good results in some cases. However, these high-level hand-designed features cannot be used for more fine-grained operations and may lead to the loss of important information, thus affecting the classification accuracy. To overcome this limitation, in this paper, we designed a novel multimodal deep learning framework for encrypted traffic classification called PEAN. PEAN uses the raw bytes and length sequence as the input, and uses the self-attention mechanism to learn the deep relationship among network packets in a biflow. Furthermore, unsupervised pre-training was introduced to enhance PEAN’s ability to characterize network packets. Experiments on a real trace set captured in a large data center demonstrate the effectiveness of PEAN, which achieves better results than the state-of-the-art methods.

A Multimodal Protein Representation Framework for Quantifying Transferability Across Biochemical Downstream Tasks.

Proteins are the building blocks of life, carrying out fundamental functions in biology. In computational biology, an effective protein representation facilitates many important biological quantifications. Most existing protein representation methods are derived from self-supervised language models designed for text analysis. Proteins, however, are more than linear sequences of amino acids. Here, a multimodal deep learning framework for incorporating ≈1 million protein sequence, structure, and functional annotation (MASSA) is proposed. A multitask learning process with five specific pretraining objectives is presented to extract a fine-grained protein-domain feature. Through pretraining, multimodal protein representation achieves state-of-the-art performance in specific downstream tasks such as protein properties (stability and fluorescence), protein-protein interactions (shs27k/shs148k/string/skempi), and protein-ligand interactions (kinase, DUD-E), while achieving competitive results in secondary structure and remote homology tasks. Moreover, a novel optimal-transport-based metric with rich geometry awareness is introduced to quantify the dynamic transferability from the pretrained representation to the related downstream tasks, which provides a panoramic view of the step-by-step learning process. The pairwise distances between these downstream tasks are also calculated, and a strong correlation between the inter-task feature space distributions and adaptability is observed.

A novel soft attention-based multi-modal deep learning framework for multi-label skin lesion classification

A novel soft attention-based multi-modal deep learning framework for multi-label skin lesion classification

DeAF: A multimodal deep learning framework for disease prediction

Multimodal deep learning models have been applied for disease prediction tasks, but difficulties exist in training due to the conflict between sub-models and fusion modules. To alleviate this issue, we propose a framework for decoupling feature alignment and fusion (DeAF), which separates the multimodal model training into two stages. In the first stage, unsupervised representation learning is conducted, and the modality adaptation (MA) module is used to align the features from various modalities. In the second stage, the self-attention fusion (SAF) module combines the medical image features and clinical data using supervised learning. Moreover, we apply the DeAF framework to predict the postoperative efficacy of CRS for colorectal cancer and whether the MCI patients change to Alzheimer’s disease. The DeAF framework achieves a significant improvement in comparison to the previous methods. Furthermore, extensive ablation experiments are conducted to demonstrate the rationality and effectiveness of our framework. In conclusion, our framework enhances the interaction between the local medical image features and clinical data, and derive more discriminative multimodal features for disease prediction. The framework implementation is available at https://github.com/cchencan/DeAF.

Deep learning for video game genre classification

In this paper, we propose a new multi-modal deep learning framework with a visual modality and a textual modality for video game genre classification. The proposed framework consists of three parts: two deeep networks for textual data and imaginary data, a feature concatenation algorithm, and then a softmax classifier. Video game covers and textual descriptions are usually the very first impression to its consumers and they often convey important information about the video games. Video game genre classification based on its cover and textual description would be utterly beneficial to many modern identification, collocation, and retrieval systems. At the same time, it is also an extremely challenging task due to the following reasons: First, there exists a wide variety of video game genres, many of which are not concretely defined. Second, video game covers vary in many different ways such as colors, styles, textual information, etc, even for games of the same genre. Third, cover designs and textual descriptions may vary due to many external factors such as country, culture, target reader populations, etc. With the growing competitiveness in the video game industry, the cover designers and typographers push the cover designs to its limit in the hope of attracting sales. The computer-based automatic video game genre classification systems become a particularly exciting research topic in recent years. The contribution of this paper is four-fold. First, we compiles a large dataset consisting of 50,000 video games from 21 genres made of cover images, description text, and title text and the genre information. Second, image-based and text-based, state-of-the-art models are evaluated thoroughly for the task of genre classification for video games. Third, we developed an efficient and scalable multi-modal framework based on both images and texts. Fourth, a thorough analysis of the experimental results is given and future works to improve the performance is suggested. The results show that the multi-modal framework outperforms the current state-of-the-art image-based or text-based models. Several challenges are outlined for this task. More efforts and resources are needed for this classification task in order to reach a satisfactory level.

Multimodal Deep-Learning Framework for Accurate Prediction of Wettability Evolution of Laser-Textured Surfaces.

The lengthy process through which laser-textured surfaces transform from hydrophilic to hydrophobic severely restricts their practical applications. Accurately predicting the wettability evolution curve is crucial; however, developing a reliable prediction model remains challenging. Herein, a data-driven multimodal deep-learning framework was developed, in which multimodal data of micro/nanostructure morphology images, composition distribution images, and time information are effectively coupled and fed into a convolutional neural network (CNN). Rich data input and in-depth data mining make the framework more robust, achieving accurate prediction of the wettability evolution curves of various typical micro/nanostructures. Additionally, accurate prediction of input images with varying magnifications and untrained laser-textured surfaces demonstrates the generalizability of the multimodal CNN framework. The visualization results of the convolution layer confirmed the rationality of the information learned by the model. Additionally, the proposed multimodal CNN framework was successfully utilized to investigate the optimization process. Further, a laser-textured surface with a shorter evolution period and a larger final contact angle was realized. The proposed multimodal CNN framework offers an efficient and cost-effective method for predicting the wettability evolution curves and exploring the optimization processes, enhancing the application potential of laser micro/nanofabrication of superhydrophobic surfaces.

A Knowledge-Graph-Based Multimodal Deep Learning Framework for Identifying Drug-Drug Interactions.

The identification of drug-drug interactions (DDIs) plays a crucial role in various areas of drug development. In this study, a deep learning framework (KGCN_NFM) is presented to recognize DDIs using coupling knowledge graph convolutional networks (KGCNs) with neural factorization machines (NFMs). A KGCN is used to learn the embedding representation containing high-order structural information and semantic information in the knowledge graph (KG). The embedding and the Morgan molecular fingerprint of drugs are then used as input of NFMs to predict DDIs. The performance and effectiveness of the current method have been evaluated and confirmed based on the two real-world datasets with different sizes, and the results demonstrate that KGCN_NFM outperforms the state-of-the-art algorithms. Moreover, the identified interactions between topotecan and dantron by KGCN_NFM were validated through MTT assays, apoptosis experiments, cell cycle analysis, and molecular docking. Our study shows that the combination therapy of the two drugs exerts a synergistic anticancer effect, which provides an effective treatment strategy against lung carcinoma. These results reveal that KGCN_NFM is a valuable tool for integrating heterogeneous information to identify potential DDIs.

DPSP: a multimodal deep learning framework for polypharmacy side effects prediction.

Because unanticipated drug-drug interactions (DDIs) can result in severe bodily harm, identifying the adverse effects of polypharmacy is one of the most important tasks in human health. Over the past few decades, computational methods for predicting the adverse effects of polypharmacy have been developed. This article presents DPSP, a framework for predicting polypharmacy side effects based on the construction of novel drug features and the application of a deep neural network to predict DDIs. In the first step, a variety of drug information is evaluated, and a feature extraction method and the Jaccard similarity are used to determine similarities between two drugs. By combining these similarities, a novel feature vector is generated for each drug. In the second step, the method predicts DDIs for specific DDI events using a multimodal framework and drug feature vectors. On three benchmark datasets, the performance of DPSP is measured by comparing its results to those of several well-known methods, such as GNN-DDI, MSTE, MDF-SA-DDI, NNPS, DDIMDL, DNN, DeepDDI, KNN, LR, and RF. DPSP outperforms these classification methods based on a variety of classification metrics. The results indicate that the use of diverse drug information is effective and efficient for identifying DDI adverse effects. The source code and datasets are available at https://github.com/raziyehmasumshah/DPSP.

Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework

The recent success of attention mechanism-driven deep models, like Vision Transformer (ViT) as one of the most representative, has intrigued a wave of advanced research to explore their adaptation to broader domains. However, current Transformer-based approaches in the remote sensing (RS) community pay more attention to single-modality data, which might lose expandability in making full use of the ever-growing multimodal Earth observation data. To this end, we propose a novel multimodal deep learning framework by extending conventional ViT with minimal modifications, abbreviated as ExViT, aiming at the task of land use and land cover classification. Unlike common stems that adopt either linear patch projection or deep regional embedder, our approach processes multimodal RS image patches with parallel branches of position-shared ViTs extended with separable convolution modules, which offers an economical solution to leverage both spatial and modality-specific channel information. Furthermore, to promote information exchange across heterogeneous modalities, their tokenized embeddings are then fused through a cross-modality attention module by exploiting pixel-level spatial correlation in RS scenes. Both of these modifications significantly improve the discriminative ability of classification tokens in each modality and thus further performance increase can be finally attained by a full tokens-based decision-level fusion module. We conduct extensive experiments on two multimodal RS benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral and light detection and ranging (LiDAR) data, and Berlin dataset with hyperspectral and synthetic aperture radar (SAR) data, to demonstrate that our ExViT outperforms concurrent competitors based on Transformer or convolutional neural network (CNN) backbones, in addition to several competitive machine learning-based models. The source codes and investigated datasets of this work will be made publicly available at https://github.com/jingyao16/ExViT.

TSCMDL: Multimodal Deep Learning Framework for Classifying Tree Species Using Fusion of 2-D and 3-D Features

Accurate tree species information is a prerequisite for forest resource management. Combining light detection and ranging (LiDAR) and image data is one main method of tree species classification. Traditional machinelearningmethods rely on expert knowledge to calculatea large number of feature parameters.Deep learning technology can directly use the original image and pointclouddata to classify tree species. However, data with different patterns require the use of different types of deeplearningmethods. In this study, a multimodal deeplearningframework (TSCMDL) that fuses 2D and 3D features was constructed and then used to combine data from multiple sources for tree species classification. This framework uses an improved version of the PointMLP model as its backbone network and uses ResNet50 and PointMLP networks to extract the image features and pointcloudfeatures, respectively. The proposed framework was tested using UAV LiDAR data and RGB orthophotos. The results showed that the accuracy of the tree species classification using the TSCMDL framework was 98.52%, which was 4.02% higher than that based on pointcloudfeatures only. In addition, when the same hyperparameters were used for training the model, the efficiency of the model training was not significantly lower than for models based on pointcloudfeatures only. The proposed multimodal deeplearningframework extracts features directly from the original data and integrates them effectively, thus avoiding manual feature screening and achieving more accurate classification. The feature extraction network used in the TSCMDL framework can be replaced by other suitable frameworks and has strong application potential.