Multimodal Hybrid Research Articles

Multi-modal hybrid hierarchical classification approach with transformers to enhance complex human activity recognition

Multi-modal hybrid hierarchical classification approach with transformers to enhance complex human activity recognition

Unprecedented mechanical wave energy absorption observed in multifunctional bioinspired architected metamaterials

In practical engineering, noise and impact hazards are pervasive, indicating the pressing demand for materials that can absorb both sound and stress wave energy simultaneously. However, the rational design of such multifunctional materials remains a challenge. Herein, inspired by cuttlebone, we present bioinspired architected metamaterials with unprecedented sound-absorbing and mechanical properties engineered via a weakly-coupled design. The acoustic elements feature heterogeneous multilayered resonators, whereas the mechanical responses are based on asymmetric cambered cell walls. These metamaterials experimentally demonstrated an average absorption coefficient of 0.80 from 1.0 to 6.0 kHz, with 77% of the data points exceeding the desired 0.75 threshold, all with a compact 21 mm thickness. An absorptance-thickness map is devised for assessing the sound-absorption efficiency. The high-fidelity microstructure-based model reveals the air friction damping mechanism, with broadband behavior attributed to multimodal hybrid resonance. Empowered by the cambered design of cell walls, metamaterials shift catastrophic failure toward a progressive deformation mode characterized by stable stress plateaus and ultrahigh specific energy absorption of 50.7 J/g—a 558.4% increase over the straight-wall design. After the deformation mechanisms are elucidated, a comprehensive research framework for burgeoning acousto-mechanical metamaterials is proposed. Overall, our study broadens the horizon for multifunctional material design.

Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion

Visual object tracking, which is primarily based on visible light image sequences, encounters numerous challenges in complicated scenarios, such as low light conditions, high dynamic ranges, and background clutter. To address these challenges, incorporating the advantages of multiple visual modalities is a promising solution for achieving reliable object tracking. However, the existing approaches usually integrate multimodal inputs through adaptive local feature interactions, which cannot leverage the full potential of visual cues, thus resulting in insufficient feature modeling. In this study, we propose a novel multimodal hybrid tracker (MMHT) that utilizes frame-event-based data for reliable single object tracking. The MMHT model employs a hybrid backbone consisting of an artificial neural network (ANN) and a spiking neural network (SNN) to extract dominant features from different visual modalities and then uses a unified encoder to align the features across different domains. Moreover, we propose an enhanced transformer-based module to fuse multimodal features using attention mechanisms. With these methods, the MMHT model can effectively construct a multiscale and multidimensional visual feature space and achieve discriminative feature modeling. Extensive experiments demonstrate that the MMHT model exhibits competitive performance in comparison with that of other state-of-the-art methods. Overall, our results highlight the effectiveness of the MMHT model in terms of addressing the challenges faced in visual object tracking tasks.

Multimodal modeling with low-dose CT and clinical information for diagnostic artificial intelligence on mediastinal tumors: a preliminary study

BackgroundDiagnosing mediastinal tumours, including incidental lesions, using low-dose CT (LDCT) performed for lung cancer screening, is challenging. It often requires additional invasive and costly tests for proper characterisation and surgical...

Multimodal approach for early prediction of COVID-19 disease using convolutional neural network

<p><span>The latest human coronavirus is COVID-19. Chest radiography imaging is essential for screening, early detection, and monitoring COVID-19 infections since the virus resides in the lungs. Classical real time reverse transcriptase polymerase chain reaction (RT-PCR) data and chest X-ray pictures will become more important for COVID-19 identification as the pandemic spreads due to their affordability, wide availability, and infection control benefits, which reduce cross-contamination. This work presents multi-modal hybrid automated approaches to classify COVID-19 illness into three clinical categories: normal, pathogenic, and COVID-19 utilising RT-PCR test data and online chest X-ray datasets. The RT-PCR and chest X-ray image datasets were processed using supervised machine learning and convolutional neural networks (CNN). Together, these measures help us separate COVID-19 patients, those with similar symptoms, and healthy persons. The author improved detection times and classification accuracy with extra tree classifier’s feature selection and openCV’s image sharpening. The proposed approaches were tested using a research dataset. The proposed methods allowed reliable COVID-19 disease categorization for clinical decision-making, with random forest (RF) classifier global precision values of 91.58% on the RT-PCR dataset and CNN model accuracy of 95.46% on improved sharpened images.</span></p>

ETFormer: An Efficient Transformer Based on Multimodal Hybrid Fusion and Representation Learning for RGB-D-T Salient Object Detection

ETFormer: An Efficient Transformer Based on Multimodal Hybrid Fusion and Representation Learning for RGB-D-T Salient Object Detection

A survey of multimodal hybrid deep learning for computer vision: Architectures, applications, trends, and challenges

In recent years, deep learning algorithms have rapidly revolutionized artificial intelligence, particularly machine learning, enabling researchers and practitioners to extend previously hand-crafted feature extraction procedures. In particular, deep learning uses adaptive learning processes to learn more complex and informative patterns from datasets of varying sizes. With the increasing availability of multimodal data streams and recent advances in deep learning algorithms, multimodal deep learning is on the rise. This requires the development of complex models that can process and analyze multimodal information in a consistent manner. However, unstructured data can come in many different forms (also known as modalities). Extracting relevant features from this data remains an ambitious goal for deep learning researchers. According to the literature, most deep learning systems consist of a single architecture (i.e., standalone deep learning). When two or more deep learning architectures are combined over multiple sensory modalities, the result is called a multimodal hybrid deep learning model. Since this research direction has received much attention in the field of deep learning, the purpose of this survey is to provide a broader overview of the topic. In this paper, we provide a comprehensive review of recent advances in multimodal hybrid deep learning, including a thorough analysis of the most commonly developed hybrid architectures. In particular, one of the main challenges in multimodal hybrid analysis is the ability of these architectures to systematically integrate cross-modal features in hybrid designs. Therefore, we propose a generic framework for multimodal hybrid learning that focuses mainly on fusion methods. We also identify trends and challenges in multimodal hybrid learning and provide insights and directions for future research. Our findings show that multimodal hybrid learning can perform well in a variety of challenging computer vision applications and tasks.

A Triplet Network Fusing Optical and SAR Images for Colored Steel Building Extraction.

The identification of colored steel buildings in images is crucial for managing the construction sector, environmental protection, and sustainable urban development. Current deep learning methods for optical remote sensing images often encounter challenges such as confusion between the roof color or shape of regular buildings and colored steel structures. Additionally, common semantic segmentation networks exhibit poor generalization and inadequate boundary regularization when extracting colored steel buildings. To overcome these limitations, we utilized the metal detection and differentiation capabilities inherent in synthetic aperture radar (SAR) data to develop a network that integrates optical and SAR data. This network, employing a triple-input structure, effectively captures the unique features of colored steel buildings. We designed a multimodal hybrid attention module in the network that discerns the varying importance of each data source depending on the context. Additionally, a boundary refinement (BR) module was introduced to extract the boundaries of the colored steel buildings in a more regular manner, and a deep supervision strategy was implemented to improve the performance of the network in the colored steel building extraction task. A BR module and deep supervision strategy were also implemented to sharpen the extraction of building boundaries, thereby enhancing the network's accuracy and adaptability. The results indicate that, compared to mainstream semantic segmentation, this method effectively enhances the precision of colored steel building detection, achieving an accuracy rate of 83.19%. This improvement marks a significant advancement in monitoring illegal constructions and supporting the sustainable development of the Beijing-Tianjin-Hebei metropolitan region.

Multi-modal hybrid modeling strategy based on Gaussian Mixture Variational Autoencoder and spatial–temporal attention: Application to industrial process prediction

The industrial process is characterized by its multi-modal nature and complex spatial and temporal correlations. Despite the fact that several multi-modal methods have been proposed, few of them can effectively extract deep multi-modal representations and the highly intricate spatial and temporal relationships. In this paper, a novel multi-modal hybrid modeling strategy (GMVAE-STA) is proposed for industrial process prediction. This strategy combines the Gaussian Mixture Variational Autoencoder (GMVAE) and the spatial–temporal attention based Gated Recurrent Unit (STA-GRU). First, the GMVAE maps the raw data to the latent space, which follows a Gaussian Mixture distribution, and the data with the highest probability in each Gaussian are identified as a mode. Then, the STA-GRU captures the complex spatial and temporal relationships within each mode and makes predictions. Experimental results on the Tennessee Eastman process and a real-world fluid catalytic cracking process demonstrate the effectiveness of mode classification and prediction of the proposed method.

Multimodal hybrid convolutional neural network based brain tumor grade classification

An abnormal growth or fatty mass of cells in the brain is called a tumor. They can be either healthy (normal) or become cancerous, depending on the structure of their cells. This can result in increased pressure within the cranium, potentially causing damage to the brain or even death. As a result, diagnostic procedures such as computed tomography, magnetic resonance imaging, and positron emission tomography, as well as blood and urine tests, are used to identify brain tumors. However, these methods can be labor-intensive and sometimes yield inaccurate results. Instead of these time-consuming methods, deep learning models are employed because they are less time-consuming, require less expensive equipment, produce more accurate results, and are easy to set up. In this study, we propose a method based on transfer learning, utilizing the pre-trained VGG-19 model. This approach has been enhanced by applying a customized convolutional neural network framework and combining it with pre-processing methods, including normalization and data augmentation. For training and testing, our proposed model used 80% and 20% of the images from the dataset, respectively. Our proposed method achieved remarkable success, with an accuracy rate of 99.43%, a sensitivity of 98.73%, and a specificity of 97.21%. The dataset, sourced from Kaggle for training purposes, consists of 407 images, including 257 depicting brain tumors and 150 without tumors. These models could be utilized to develop clinically useful solutions for identifying brain tumors in CT images based on these outcomes.

Automated diagnosis of pancreatic mucinous and serous cystic neoplasms with modality-fusion deep neural network using multi-modality MRIs.

Pancreatic cystic neoplasms are increasingly diagnosed with the development of medical imaging technology and people's self-care awareness. However, two of their sub-types, serous cystic neoplasms (SCN) and mucinous cystic neoplasms (MCN), are often misclassified from each other. Because SCN is primarily benign and MCN has a high rate of malignant transformation. Distinguishing SCN and MCN is challenging and essential. MRIs have many different modalities, complete with SCN and MCN diagnosis information. With the help of an artificial intelligence-based algorithm, we aimed to propose a multi-modal hybrid deep learning network that can efficiently diagnose SCN and MCN using multi-modality MRIs. A cross-modal feature fusion structure was innovatively designed, combining features of seven modalities to realize the classification of SCN and MCN. 69 Patients with multi-modalities of MRIs were included, and experiments showed performances of every modality. The proposed method with the optimized settings outperformed all other techniques and human radiologists with high accuracy of 75.07% and an AUC of 82.77%. Besides, the proposed disentanglement method outperformed other fusion methods, and delayed contrast-enhanced T1-weighted MRIs proved most valuable in diagnosing SCN and MCN. Through the use of a contemporary artificial intelligence algorithm, physicians can attain high performance in the complex challenge of diagnosing SCN and MCN, surpassing human radiologists to a significant degree.

An adaptive multi-modal hybrid model for classifying thyroid nodules by combining ultrasound and infrared thermal images

BackgroundTwo types of non-invasive, radiation-free, and inexpensive imaging technologies that are widely employed in medical applications are ultrasound (US) and infrared thermography (IRT). The ultrasound image obtained by ultrasound imaging primarily expresses the size, shape, contour boundary, echo, and other morphological information of the lesion, while the infrared thermal image obtained by infrared thermography imaging primarily describes its thermodynamic function information. Although distinguishing between benign and malignant thyroid nodules requires both morphological and functional information, present deep learning models are only based on US images, making it possible that some malignant nodules with insignificant morphological changes but significant functional changes will go undetected.ResultsGiven the US and IRT images present thyroid nodules through distinct modalities, we proposed an Adaptive multi-modal Hybrid (AmmH) classification model that can leverage the amalgamation of these two image types to achieve superior classification performance. The AmmH approach involves the construction of a hybrid single-modal encoder module for each modal data, which facilitates the extraction of both local and global features by integrating a CNN module and a Transformer module. The extracted features from the two modalities are then weighted adaptively using an adaptive modality-weight generation network and fused using an adaptive cross-modal encoder module. The fused features are subsequently utilized for the classification of thyroid nodules through the use of MLP. On the collected dataset, our AmmH model respectively achieved 97.17% and 97.38% of F1 and F2 scores, which significantly outperformed the single-modal models. The results of four ablation experiments further show the superiority of our proposed method.ConclusionsThe proposed multi-modal model extracts features from various modal images, thereby enhancing the comprehensiveness of thyroid nodules descriptions. The adaptive modality-weight generation network enables adaptive attention to different modalities, facilitating the fusion of features using adaptive weights through the adaptive cross-modal encoder. Consequently, the model has demonstrated promising classification performance, indicating its potential as a non-invasive, radiation-free, and cost-effective screening tool for distinguishing between benign and malignant thyroid nodules. The source code is available at https://github.com/wuliZN2020/AmmH.

HF-DDI: Predicting Drug-Drug Interaction Events Based on Multimodal Hybrid Fusion.

Drug-drug interactions (DDIs) can have a significant impact on patient safety and health. Predicting potential DDIs before administering drugs to patients is a critical step in drug development and can help prevent adverse drug events. In this study, we propose a novel method called HF-DDI for predicting DDI events based on various drug features, including molecular structure, target, and enzyme information. Specifically, we design our model with both early fusion and late fusion strategies and utilize a score calculation module to predict the likelihood of interactions between drugs. Our model was trained and tested on a large data set of known DDIs, achieving an overall accuracy of 0.948. The results suggest that incorporating multiple drug features can improve the accuracy of DDI event prediction and may be useful for improving drug safety and patient outcomes.

A multimodal hybrid parallel network intrusion detection model

With the rapid growth of Internet data traffic, the means of malicious attack become more diversified. The single modal intrusion detection model cannot fully exploit the rich feature information in the massive network traffic data, resulting in unsatisfactory detection results. To address this issue, this paper proposes a multimodal hybrid parallel network intrusion detection model (MHPN). The proposed model extracts network traffic features from two modalities: the statistical information of network traffic and the original load of traffic, and constructs appropriate neural network models for each modal information. Firstly, a two-branch convolutional neural network is combined with Long Short-Term Memory (LSTM) network to extract the spatio-temporal feature information of network traffic from the original load mode of traffic, and a convolutional neural network is used to extract the feature information of traffic statistics. Then, the feature information extracted from the two modalities is fused and fed to the CosMargin classifier for network traffic classification. The experimental results on the ISCX-IDS 2012 and CIC-IDS-2017 datasets show that the MHPN model outperforms the single-modal models and achieves an average accuracy of 99.98 % . The model also demonstrates strong robustness and a positive sample recognition rate.

The intercultural communication competence improvement for pre-service CSL teachers: A blended learning method based on SVVR

BACKGROUND: With the growing global influence of new coronary pneumonia and an increasing number of successful education case, spherical video-based virtual reality (SVVR) has arisen as an application for effective integration with teaching and learning. OBJECTIVE: This is a mixed-methods study on SVVR-based training for pre-service CSL teachers to more effectively develop their cross-cultural communication and interaction abilities. METHODS: An experimental strategy was adopted in this work. First, the processing of VR technology for designed TVET courses, including practical training courses, had been completed; The second experiment was a group experiment of one month; Finally, the hypotheses were tested by an independent sample t-test. RESULTS: Based on experiment data of the learners, this study build a multi-modal hybrid training and recognition model and confirm hybrid approach effectively improves the intercultural communication abilities of pre-service CSL teachers. CONCLUSIONS: The findings of this study confirm that information technology plays an important role in pre-service CSL teacher training, and teachers should incorporate more information technology, design more hands-on interactive activities and scenarios, and take into account the individual differences of learners in the teaching process. Several policy proposals are offered to ensure that ICT is better integrated into classroom instruction.

MRLab: Virtual-Reality Fusion Smart Laboratory Based on Multimodal Fusion

During the COVID-19 pandemic, online classes became the only option for many students. The main challenge for these classes was conducting risky and complex chemical or biological experiments in a domestic environment. To address this challenge, a smart experiment system called MRLab was developed. MRLab used wearables such as a smart glove and head-mounted device to record sensory data and a multimodal hybrid fusion model GVVS to interpret the user’s experimental intent, which essentially transforms the user’s abstract behavioral actions into a probabilistic set of experimental intent that can be computed. Different experiments in MRLab used different libraries of experimental intents. The SrNet model in GVVS was used to estimate the probability of the user’s gesture behavior generated from the smart glove, while the SIPA algorithm compared speech information entered during the experiment with the experimental intent library to estimate the probability of the user’s intent. At the same time, the scene visual channel monitored the information about the object the user intended to operate, with the SVF algorithm computing the probability of the intended object in real-time. The results from ANOVA and post-hoc comparative testing conducted on 21 volunteers revealed that MRLab outperformed other experiment modes, including WEB, AR, and VR, with a higher intention understanding rate, efficiency, and user satisfaction. Therefore, MRLab proved to be a useful alternative to traditional physics laboratory experiments during the pandemic, along with being an additional teaching tool for remote learning purposes.

Using Hybrid Bioimaging Analysis for Rare Neurological Diseases

Hybrid PET/MRI is a promising new approach for simultaneously investigating the relationships between intrinsic metabolic and functional brain changes. In this abstract, we present an analysis workflow for longitudinal PET/MRI studies in rare neurological diseases. We envision that software tools based on the presented workflow can be used to evaluate progressive disease-related volumetric, functional, and metabolic alterations. Furthermore, it could be useful for characterize degenerative pathologies with fast dynamics. Our main contribution is the use of multimodal hybrid PET/MRI data fusion to identify potential markers of disease progression, which could support physicians in future trials to test disease-modifying drugs.

Brain-inspired multimodal hybrid neural network for robot place recognition.

Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi-neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.

Medical image segmentation based on self-supervised hybrid fusion network.

Automatic segmentation of medical images has been a hot research topic in the field of deep learning in recent years, and achieving accurate segmentation of medical images is conducive to breakthroughs in disease diagnosis, monitoring, and treatment. In medicine, MRI imaging technology is often used to image brain tumors, and further judgment of the tumor area needs to be combined with expert analysis. If the diagnosis can be carried out by computer-aided methods, the efficiency and accuracy will be effectively improved. Therefore, this paper completes the task of brain tumor segmentation by building a self-supervised deep learning network. Specifically, it designs a multi-modal encoder-decoder network based on the extension of the residual network. Aiming at the problem of multi-modal feature extraction, the network introduces a multi-modal hybrid fusion module to fully extract the unique features of each modality and reduce the complexity of the whole framework. In addition, to better learn multi-modal complementary features and improve the robustness of the model, a pretext task to complete the masked area is set, to realize the self-supervised learning of the network. Thus, it can effectively improve the encoder's ability to extract multi-modal features and enhance the noise immunity. Experimental results present that our method is superior to the compared methods on the tested datasets.

A multimodal hybrid stochastic-based deterministic ARFIMA model for the sustainable analysis of COVID-19 pandemic

The revise abstract is given as follows: The rapid emergence of the super-spreader COVID-19 with severe economic calamities with devastating social impact worldwide created the demand for effective research on the spread dynamics of the disease to combat and create surveillance systems on a global scale. In this study, a novel hybrid Deterministic Autoregressive Fractional Integral Moving Average (ARFIMA) model is presented to forecast the bimodal COVID-19 transmission dynamics. The heterogeneity of multimodal behavior of the COVID-19 pandemic in Pakistan is modeled by a hybrid paradigm, in which a deterministic pattern is combined with the ARFIMA model to absorb the inherent chaotic pattern of the pandemic spread. The fractional fluctuation of the real epidemic system is effectively taken as a paradigm by stochastic type improved the deterministic model and ARFIMA process. Special transformations are also introduced to enhance the convergent rate of the bimodal paradigm in deterministic modeling. The outcome of the improved deterministic model is combined with the ARFIMA model is evaluated on the spread pattern of pandemic data in Pakistan for the next 30 days. The performance-indices of the hybrid-model based on Relative-Errors and RMSE statistics confirmed the effectiveness of the proposed paradigm for long-term epidemic modeling compared to other classical and machine learning algorithms.