Multimodal Framework Research Articles

EGeRepDR: An enhanced genetic-based representation learning for drug repurposing using multiple biomedical sources

MotivationDrug repurposing (DR) is an imminent approach for identifying novel therapeutic indications for the available drugs and discovering novel drugs for previously untreatable diseases. Nowadays, DR has major attention in the pharmaceutical industry due to the high cost and time of launching new drugs to the market through traditional drug development. DR task majorly depends on genetic information since the drugs revert the modified Gene Expression (GE) of diseases to normal. Many of the existing studies have not considered the genetic importance of predicting the potential candidates. MethodWe proposed a novel multimodal framework that utilizes genetic aspects of drugs and diseases such as genes, pathways, gene signatures, or expression to enhance the performance of DR using various data sources. Firstly, the heterogeneous biological network (HBN) is constructed with three types of nodes namely drug, disease, and gene, and 4 types of edges similarities (drug, gene, and disease), drug-gene, gene-disease, and drug-disease. Next, a modified graph auto-encoder (GAE*) model is applied to learn the representation of drug and disease nodes using the topological structure and edge information. Secondly, the HBN is enhanced with the information extracted from biomedical literature and ontology using a novel semi-supervised pattern embedding-based bootstrapping model and novel DR perspective representation learning respectively to improve the prediction performance. Finally, our proposed system uses a neural network model to generate the probability score of drug-disease pairs. ResultsWe demonstrate the efficiency of the proposed model on various datasets and achieved outstanding performance in 5-fold cross-validation (AUC = 0.99, AUPR = 0.98). Further, we validated the top-ranked potential candidates using pathway analysis and proved that the known and predicted candidates share common genes in the pathways.

An end-to-end multimodal 3D CNN framework with multi-level features for the prediction of mild cognitive impairment

In recent years, deep learning methods based on brain image have been used for the diagnosis of cognitive impairment-related disorders. With the development of neuroimaging techniques, multi-modality image such as structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) reflect structural and functional information of the brain respectively, and provide more techniques for the diagnosis of cognitive impairment diseases. Combining these complementary image features can lead to more accurate diagnostic assessments compared to using a single modality. Therefore, how to effectively combine multi-modality image features to realize the diagnosis of cognitive impairment disease needs to be further explored. In this work, we propose an end-to-end multimodal 3D CNN framework based on ResNet architecture, which integrates multi-level features obtained under the role of attention mechanisms to better capture subtle differences among brain images, and achieves remarkable diagnostic performance through spatial pyramid pooling strategy and effective fusion of multi-modality features. In this process, we demonstrate that the multimodal framework is more effective by means of non-shared parameters for multi-modality features learning. Moreover, the visualized attention maps show that our model can focus on important brain regions relevant to disease diagnosis. The experimental results demonstrated that our method improved the diagnostic performance in AD diagnosis and MCI conversion prediction by 6.37 % and 3.51 % compared to the single modality, and it also outperformed some recent state-of-the-art multimodal methods. Especially in AD diagnosis achieved an average accuracy of 94.61 %, which provides a more feasible technology for diagnostic assessment of patients with AD.

Feature importance in the age of explainable AI: Case study of detecting fake news & misinformation via a multi-modal framework

In recent years, fake news has become a global phenomenon due to its explosive growth and ability to leverage multimedia content to manipulate user opinions. Fake news is created by manipulating images, text, audio, and videos, particularly on social media, and the proliferation of such disinformation can trigger detrimental societal effects. False forwarded messages can have a devastating impact on society, spreading propaganda, inciting violence, manipulating public opinion, and even influencing elections. A major shortcoming of existing fake news detection methods is their inability to simultaneously learn and extract features from two modalities and train models with shared representations of multimodal (textual and visual) information. Feature engineering is a critical task in the fake news detection model's machine learning (ML) development process. For ML models to be explainable and trusted, feature engineering should describe how many features used in the ML models contribute to making more accurate predictions. Feature engineering, which plays an important role in the development of an explainable AI system by shaping the features used in the ML models, is an interconnected concept with explainable AI as it affects the model's interpretability. In the research, we develop a fake news detector model in which we (1) identify several textual and visual features that are associated with fake or credible news; specifically, we extract features from article titles, contents, and, top images; (2) investigate the role of all multimodal features (content, emotions and manipulation-based) and combine the cumulative effects using the feature engineering that represent the behavior of fake news propagators; and (3) develop a model to detect disinformation on benchmark multimodal datasets consisting of text and images. We conduct experiments on several real-world multimodal fake news datasets, and our results show that on average, our model outperforms existing single-modality methods by large margins that do not use any feature optimization techniques.

“We Learn Through Mistakes”: Perspectives of Social Media Creators on Copyright Moderation in the European Union

Artists promoting their cultural production on social media platforms must navigate an algorithmic environment that involves visibility, content distribution, and growing algorithmic copyright moderation. The latter was experienced by EU-based social media creators when Article 17 of the Copyright in the Digital Single Market Directive was enforced in the EU (Directive [EU] 2019/790, 2019). This research focuses on the understanding and experiences of such creators regarding copyright content moderation of their work on social media platforms. Semi-structured interviews were conducted with 14 artists from EU countries, who were surveyed on digitization and digital access to cultural content. A multimodal framework was employed to analyze the copyright governance of creative practices and products, focusing on the regulative (i.e., the adoption of Article 17), normative (assumptions about legitimate and illegitimate behavior in specific communities), and discursive dimensions, as well as the influence of technological affordance on creative work. The findings contribute to the fields of cultural production and platform governance, as well as having policy implications. The anticipation of punishments from platforms due to copyright-related fears and the inability to use appeal processes directly influenced cultural production. Most interviewees employed self-censorship, avoidance, and content adjustment in their creative work before posting on social media platforms.

Collocated Clothing Synthesis with GANs Aided by Textual Information: A Multi-Modal Framework

Synthesizing realistic images of fashion items which are compatible with given clothing images, as well as conditioning on multiple modalities, brings novel and exciting applications together with enormous economic potential. In this work, we propose a multi-modal collocation framework based on generative adversarial network (GAN) for synthesizing compatible clothing images. Given an input clothing item that consists of an image and a text description, our model works on synthesizing a clothing image which is compatible with the input clothing, as well as being guided by a given text description from the target domain. Specifically, a generator aims to synthesize realistic and collocated clothing images relying on image- and text-based latent representations learned from the source domain. An auxiliary text representation from the target domain is added for supervising the generation results. In addition, a multi-discriminator framework is carried out to determine compatibility between the generated clothing images and the input clothing images, as well as visual-semantic matching between the generated clothing images and the targeted textual information. Extensive quantitative and qualitative results demonstrate that our model substantially outperforms state-of-the-art methods in terms of authenticity, diversity, and visual-semantic similarity between image and text.

A Unified Multimodal Interface for the RELAX High-Payload Collaborative Robot.

This manuscript introduces a mobile cobot equipped with a custom-designed high payload arm called RELAX combined with a novel unified multimodal interface that facilitates Human-Robot Collaboration (HRC) tasks requiring high-level interaction forces on a real-world scale. The proposed multimodal framework is capable of combining physical interaction, Ultra Wide-Band (UWB) radio sensing, a Graphical User Interface (GUI), verbal control, and gesture interfaces, combining the benefits of all these different modalities and allowing humans to accurately and efficiently command the RELAX mobile cobot and collaborate with it. The effectiveness of the multimodal interface is evaluated in scenarios where the operator guides RELAX to reach designated locations in the environment while avoiding obstacles and performing high-payload transportation tasks, again in a collaborative fashion. The results demonstrate that a human co-worker can productively complete complex missions and command the RELAX mobile cobot using the proposed multimodal interaction framework.

A noise-immune and attention-based multi-modal framework for short-term traffic flow forecasting

A noise-immune and attention-based multi-modal framework for short-term traffic flow forecasting

Prediction of Protein-Protein Interactions Using Vision Transformer and Language Model.

The knowledge of protein-protein interaction (PPI) helps us to understand proteins' functions, the causes and growth of several diseases, and can aid in designing new drugs. The majority of existing PPI research has relied mainly on sequence-based approaches. With the availability of multi-omics datasets (sequence, 3D structure) and advancements in deep learning techniques, it is feasible to develop a deep multi-modal framework that fuses the features learned from different sources of information to predict PPI. In this work, we propose a multi-modal approach utilizing protein sequence and 3D structure. To extract features from the 3D structure of proteins, we use a pre-trained vision transformer model that has been fine-tuned on the structural representation of proteins. The protein sequence is encoded into a feature vector using a pre-trained language model. The feature vectors extracted from the two modalities are fused and then fed to the neural network classifier to predict the protein interactions. To showcase the effectiveness of the proposed methodology, we conduct experiments on two popular PPI datasets, namely, the human dataset and the S. cerevisiae dataset. Our approach outperforms the existing methodologies to predict PPI, including multi-modal approaches. We also evaluate the contributions of each modality by designing uni-modal baselines. We perform experiments with three modalities as well, having gene ontology as the third modality.

Neuroimaging and machine learning for studying the pathways from mild cognitive impairment to alzheimer’s disease: a systematic review

BackgroundThis systematic review synthesizes the most recent neuroimaging procedures and machine learning approaches for the prediction of conversion from mild cognitive impairment to Alzheimer’s disease dementia.MethodsWe systematically searched PubMed, SCOPUS, and Web of Science databases following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) systematic review guidelines.ResultsOur search returned 2572 articles, 56 of which met the criteria for inclusion in the final selection. The multimodality framework and deep learning techniques showed potential for predicting the conversion of MCI to AD dementia.ConclusionFindings of this systematic review identified that the possibility of using neuroimaging data processed by advanced learning algorithms is promising for the prediction of AD progression. We also provided a detailed description of the challenges that researchers are faced along with future research directions. The protocol has been registered in the International Prospective Register of Systematic Reviews– CRD42019133402 and published in the Systematic Reviews journal.

Deep learning-enabled 3D multimodal fusion of cone-beam CT and intraoral mesh scans for clinically applicable tooth-bone reconstruction

High-fidelity three-dimensional (3D) models of tooth-bone structures are valuable for virtual dental treatment planning; however, they require integrating data from cone-beam computed tomography (CBCT) and intraoral scans (IOS) using methods that are either error-prone or time-consuming. Hence, this study presents Deep Dental Multimodal Fusion (DDMF), an automatic multimodal framework that reconstructs 3D tooth-bone structures using CBCT and IOS. Specifically, the DDMF framework comprises CBCT and IOS segmentation modules as well as a multimodal reconstruction module with novel pixel representation learning architectures, prior knowledge-guided losses, and geometry-based 3D fusion techniques. Experiments on real-world large-scale datasets revealed that DDMF achieved superior segmentation performance on CBCT and IOS, achieving a 0.17mm average symmetric surface distance (ASSD) for 3D fusion with a substantial processing time reduction. Additionally, clinical applicability studies have demonstrated DDMF's potential for accurately simulating tooth-bone structures throughout the orthodontic treatment process.

Towards modern post-coma care based on neuroscientific evidence.

Understanding the mechanisms underlying human consciousness is pivotal to improve the prognostication and treatment of severely brain-injured patients. Consciousness remains an elusive concept and the identification of its neural correlates is an active subject of research, however recent neuroscientific advances have allowed scientists to better characterize disorders of consciousness. These breakthroughs question the historical nomenclature and our current management of post-comatose patients. This review examines the contribution of consciousness neurosciences to the current clinical management of severe brain injury. It investigates the major impact of consciousness disorders on healthcare systems, the scientific frameworks employed to identify their neural correlates and how evidence-based data from neuroimaging research have reshaped the landscape of post-coma care in recent years. Our increased ability to detect behavioral and neurophysiological signatures of consciousness has led to significant changes in taxonomy and clinical practice. We advocate for a multimodal framework for the management of severely brain-injured patients based on precision medicine and evidence-based decisions, integrating epidemiology, health economics and neuroethics. Major progress in brain imaging and clinical assessment have opened the door to a new era of post-coma care based on standardized neuroscientific evidence. We highlight its implications in clinical applications and call for improved collaborations between researchers and clinicians to better translate findings to the bedside.

Deep Multi-Modal Network Based Automated Depression Severity Estimation

Depression is a severe mental illness that impairs a person's capacity to function normally in personal and professional life. The assessment of depression usually requires a comprehensive examination by an expert professional. Recently, machine learning-based automatic depression assessment has received considerable attention for a reliable and efficient depression diagnosis. Various techniques for automated depression detection were developed; however, certain concerns still need to be investigated. In this work, we propose a novel deep multi-modal framework that effectively utilizes facial and verbal cues for an automated depression assessment. Specifically, we first partition the audio and video data into fixed-length segments. Then, these segments are fed into the Spatio-Temporal Networks as input, which captures both spatial and temporal features as well as assigns higher weights to the features that contribute most. In addition, Volume Local Directional Structural Pattern (VLDSP) based dynamic feature descriptor is introduced to extract the facial dynamics by encoding the structural aspects. Afterwards, we employ the Temporal Attentive Pooling (TAP) approach to summarize the segment-level features for audio and video data. Finally, the multi-modal factorized bilinear pooling (MFB) strategy is applied to fuse the multi-modal features effectively. An extensive experimental study reveals that the proposed method outperforms state-of-the-art approaches.

Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA

In this paper, we propose a novel multi-modal framework for Scene Text Visual Question Answering (STVQA), which requires models to read scene text in images for question answering. Apart from text or visual objects, which could exist independently, scene text naturally links text and visual modalities together by conveying linguistic semantics while being a visual object in an image simultaneously. Different to conventional STVQA models which take the linguistic semantics and visual semantics in scene text as two separate features, in this paper, we propose a paradigm of "Locate Then Generate" (LTG), which explicitly unifies this two semantics with the spatial bounding box as a bridge connecting them. Specifically, at first, LTG locates the region in an image that may contain the answer words with an answer location module (ALM) consisting of a region proposal network and a language refinement network, both of which can transform to each other with one-to-one mapping via the scene text bounding box. Next, given the answer words selected by ALM, LTG generates a readable answer sequence with an answer generation module (AGM) based on a pre-trained language model. As a benefit of the explicit alignment of the visual and linguistic semantics, even without any scene text based pre-training tasks, LTG can boost the absolute accuracy by +6.06% and +6.92% on the TextVQA dataset and the ST-VQA dataset respectively, compared with a non-pre-training baseline. We further demonstrate that LTG effectively unifies visual and text modalities through the spatial bounding box connection, which is underappreciated in previous methods.

Intersemiotic Shifts in the Subtitle Translation of Journey to the West via a Multimodal Framework

Journey to the West is one of the four famous Chinese classical works and it was reproduced into TV drama both at home and abroad, which has aroused great concern. However, current studies mainly compare the English translated versions with Chinese one, focusing on the translation strategies for the verbal texts or on the cultural level. Therefore, this study explores different kinds of intersemiotic shifts in the subtitle translation of Journey to the West and reasons that have caused these intersemiotic shifts via a multimodal analytical framework. Ten episodes chosen according to the plots will be analyzed. Findings show that there are mainly six kinds of intersemiotic shifts in the Journey to the West. Image and para-/non-verbal modes can supplement the meaning expressed in the verbal texts.

Enhancing Human-Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition.

With the increasing presence of robots in our daily lives, it is crucial to design interaction interfaces that are natural, easy to use and meaningful for robotic tasks. This is important not only to enhance the user experience but also to increase the task reliability by providing supplementary information. Motivated by this, we propose a multi-modal framework consisting of multiple independent modules. These modules take advantage of multiple sensors (e.g., image, sound, depth) and can be used separately or in combination for effective human-robot collaborative interaction. We identified and implemented four key components of an effective human robot collaborative setting, which included determining object location and pose, extracting intricate information from verbal instructions, resolving user(s) of interest (UOI), and gesture recognition and gaze estimation to facilitate the natural and intuitive interactions. The system uses a feature-detector-descriptor approach for object recognition and a homography-based technique for planar pose estimation and a deep multi-task learning model to extract intricate task parameters from verbal communication. The user of interest (UOI) is detected by estimating the facing state and active speakers. The framework also includes gesture detection and gaze estimation modules, which are combined with a verbal instruction component to form structured commands for robotic entities. Experiments were conducted to assess the performance of these interaction interfaces, and the results demonstrated the effectiveness of the approach.

Identifying multimodal misinformation leveraging novelty detection and emotion recognition.

With the growing presence of multimodal content on the web, a specific category of fake news is rampant on popular social media outlets. In this category of fake online information, real multimedia contents (images, videos) are used in different but related contexts with manipulated texts to mislead the readers. The presence of seemingly non-manipulated multimedia content reinforces the belief in the associated fabricated textual content. Detecting this category of misleading multimedia fake news is almost impossible without relevance to any prior knowledge. In addition to this, the presence of highly novel and emotion-invoking contents can fuel the rapid dissemination of such fake news. To counter this problem, in this paper, we first introduce a novel multimodal fake news dataset that includes background knowledge (from authenticate sources) of the misleading articles. Second, we design a multimodal framework using Supervised Contrastive Learning (SCL) based novelty detection and Emotion Prediction tasks for fake news detection. We perform extensive experiments to reveal that our proposed model outperforms the state-of-the-art (SOTA) models.

Depression Monitoring System via Social Media Data using Machine Learning frameworkk

Abstract: Stress and Depression is one of the most widely recognized and handicapping mental issue that relevantly affects society. Automatic health monitoring systems could be crucial and important to improve depression and stress detection system using social networking. Sentiment Analysis alludes to the utilization of natural language processing and content mining approaches planning to recognize feeling or opinion. Full of feeling Computing is the examination and advancement of frameworks and gadgets that can perceive, decipher, process, and mimic human effects. Sentiment Analysis and deep learning techniques could give powerful algorithms and frameworks to a target appraisal and observing of mental issue and, specifically of depression and stress. In this paper, the application of sentiment analysis and deep learning methodologies to depression and stress detection and monitoring are discussed. In addition, a fundamental plan of an incorporated multimodal framework for stress and depression checking, that incorporates estimation investigation and full of feeling processing strategies, is proposed. In particular, the paper traces the fundamental issues and moves comparative with the structure of such a framework.

Learning a holistic and comprehensive code representation for code summarization

Code summarization is the task of describing the function of code snippets in natural language, which benefits program comprehension and boosts software productivity. Despite lots of effort made by previous studies, existing models are not comprehensive enough to represent code, and yet the literature does not consider to model source code with API usage from a holistic perspective. To this end, this paper proposes a novel multi-modal code summarization approach called HCCS (Learning a Holistic and Comprehensive code representation for Code Summarization). We first design a neural network based on the graph attention mechanism to encode API Context Graph (ACG), which highlights holistic information of source code. Then, a multi-modal framework with a tree encoder for Abstract Syntax Tree (AST) and a code encoder for code tokens is incorporated to learn a more comprehensive code representation. Afterwards, we propose a fusing layer to integrate the encodings, which are then passed to a joint-decoder to generate summaries. The experimental results show that HCCS achieves better performance than the state of the arts (i.e., HCCS scores 9.5% higher in terms of BLEU metric and 11.46% in terms of BERTScore). As a result, HCCS is an effective approach to generate high-quality summaries.

The Power of ECG in Semi-Automated Seizure Detection in Addition to Two-Channel behind-the-Ear EEG.

Long-term home monitoring of people living with epilepsy cannot be achieved using the standard full-scalp electroencephalography (EEG) coupled with video. Wearable seizure detection devices, such as behind-the-ear EEG (bte-EEG), offer an unobtrusive method for ambulatory follow-up of this population. Combining bte-EEG with electrocardiography (ECG) can enhance automated seizure detection performance. However, such frameworks produce high false alarm rates, making visual review necessary. This study aimed to evaluate a semi-automated multimodal wearable seizure detection framework using bte-EEG and ECG. Using the SeizeIT1 dataset of 42 patients with focal epilepsy, an automated multimodal seizure detection algorithm was used to produce seizure alarms. Two reviewers evaluated the algorithm's detections twice: (1) using only bte-EEG data and (2) using bte-EEG, ECG, and heart rate signals. The readers achieved a mean sensitivity of 59.1% in the bte-EEG visual experiment, with a false detection rate of 6.5 false detections per day. Adding ECG resulted in a higher mean sensitivity (62.2%) and a largely reduced false detection rate (mean of 2.4 false detections per day), as well as an increased inter-rater agreement. The multimodal framework allows for efficient review time, making it beneficial for both clinicians and patients.

A Multimodal framework to develop discursive skills in engineering

Multimodality in assignments is increasingly common in higher education, thereby requiring students to demonstrate competency in the employment of multiple modes to communicate. Specifically, the field of engineering communication relies on multimodal resources to construct meaning and convey information. This article describes the integration of multimodality in a second-year software design and communication course assignment. In this assignment, students read the text, Made to Stick that enlists six principles of effective communication strategies and are expected to apply these communicative principles to a tech talk video. Students then produce a written argument which evaluates how effectively the presenter employed Made to Stick principles. Through engagement with multimodal resources, students are afforded the opportunity to practice and develop three skills of application, analysis and argumentation.