Multimodal Detection Research Articles

Multimodal Religiously Hateful Social Media Memes Classification Based on Textual and Image Data

Multimodal hateful social media meme detection is an important and challenging problem in the vision-language domain. Recent studies show high accuracy for such multimodal tasks due to datasets that provide better joint multimodal embedding to narrow the semantic gap. Religiously hateful meme detection is not extensively explored among published datasets. While there is a need for higher accuracy on religiously hateful memes, deep learning–based models often suffer from inductive bias. This issue is addressed in this work with the following contributions. First, a religiously hateful memes dataset is created and published publicly to advance hateful religious memes detection research. Over 2000 meme images are collected with their corresponding text. The proposed approach compares and fine-tunes VisualBERT pre-trained on the Conceptual Caption (CC) dataset for the downstream classification task. We also extend the dataset with the Facebook hateful memes dataset. We extract visual features using ResNeXT-152 Aggregated Residual Transformations–based Masked Regions with Convolutional Neural Networks (R-CNN) and Bidirectional Encoder Representations from Transformers (BERT) uncased for textual encoding for the early fusion model. We use the primary evaluation metric of an Area Under the Operator Characters Curve (AUROC) to measure model separability. Results show that the proposed approach has a higher AUROC score of 78%, proving the model’s higher separability performance and an accuracy of 70%. It shows comparatively superior performance considering dataset size and against ensemble-based machine learning approaches.

Enhanced Feature Representation for Multimodal Fake News Detection Using Localized Fine-Tuning of Improved BERT and VGG-19 Models

Enhanced Feature Representation for Multimodal Fake News Detection Using Localized Fine-Tuning of Improved BERT and VGG-19 Models

Multi-modal data novelty detection with adversarial autoencoders

Novelty detection is usually defined as the identification of new or abnormal objects (outliers) from the normal ones (inliers), which has wide potential applications including instrument fault, credit card theft warning, and disease diagnosis in the real world. Most of the existing researches focus on the novelty detection for single-modal data, which may fail to provide an accurate and reliable decision because the single-modal data sometimes cannot fully reflect the actual condition of the problem. However, research on the novelty detection from multiple sources is limited. This study developed an end-to-end deep learning architecture of novelty detection for multi-modal data based on adversarial autoencoders (AAE), which requires no input data of the novelty class during the training process. The proposed model consists of three deep networks, which are trained to compete with each other, collaborate to understand the underlying concept in the normal class, and thus classify the testing samples. Among the three networks, two of them work as the novelty detectors, and the other one called the generator will support the two novelty detectors by enhancing the inliers and distorting the outliers. In addition, a pseudo novelty mechanism is developed to expand the groups of adversarial training with the aim to greatly improve the precision and robustness of the whole model. Then the proposed model is employed for the novelty detection based on the multi-modal data that is composed of images, audios, videos, and texts, etc. The experimental results on the PKU FG-XMedia dataset and the MSR-VTT dataset reveal that our proposed method learns the normal class effectively and is superior to the baseline and state-of-the-art methods.

Enhancing multimodal depression detection with intra- and inter-sample contrastive learning

Multimodal depression detection (MDD) has garnered significant interest in recent years. Current methods typically integrate multimodal information within samples to distinguish positive from negative samples, but they often neglect the relationships between samples. Despite similarities within the same class, individual variations exist. By leveraging these relationships, we can provide supervision signals for both inter- and intra-class samples, thereby enhancing the discriminative power of user representations. Inspired by this observation, we introduce IISFD, a novel approach that concurrently exploits intra-sample contrastive learning and inter-sample contrastive learning with hard negative sampling. This method comprehensively considers information both within individual samples and across samples. Specifically, we decompose the multimodal inputs of each sample, including audio, vision and text, into modality-common features and modality-specific features. To obtain better decomposed feature representations, we integrate intra-sample contrastive learning and inter-sample contrastive learning with hard negative sampling. Additionally, detailed modal information is obtained through unimodal reconstruction. By passing the decomposed features through a carefully designed multimodal fusion module, we obtain more discriminative user representations. Experimental results on two publicly available datasets demonstrate the superiority of our model, highlighting its effectiveness in leveraging both intra- and inter-sample information for enhanced MDD.

CPROS: A Multimodal Decision-Level Fusion Detection Method Based on Category Probability Sets

CPROS: A Multimodal Decision-Level Fusion Detection Method Based on Category Probability Sets

Vae-Clip: Unveiling Deception through Cross-Modal Models and Multi-Feature Integration in Multi-Modal Fake News Detection

Vae-Clip: Unveiling Deception through Cross-Modal Models and Multi-Feature Integration in Multi-Modal Fake News Detection

DPSG: Dynamic Propagation Social Graphs for multi-modal fake news detection

Fake news can mislead public perception and have a negative impact on people’s lives. With the increase of multi-modal news content in social media, multi-modal fake news detection has received extensive attention from researchers. However, existing methods face difficulties in capturing temporal intensity and deep social graph representations. Considering the multi-modality of news content and the relationship between entities, this paper proposes a multi-modal fake news detection model based on dynamic propagation social graph (DPSG). Specifically, it first utilizes the temporal penetration fusion block to perform multi-modal dynamic fusion on various types of nodes in the propagation sequence, and then employs dynamic signed weighting to enhance the social graph features of the propagation graph. Finally, the Transformer captures news propagation patterns and fuses multi-domain information. Experimental results on three datasets demonstrate the effectiveness of the proposed model on the fake news detection task.

RCRFNet: Enhancing Object Detection with Self-Supervised Radar-Camera Fusion and Open-Set Recognition.

Robust object detection in complex environments, poor visual conditions, and open scenarios presents significant technical challenges in autonomous driving. These challenges necessitate the development of advanced fusion methods for millimeter-wave (mmWave) radar point cloud data and visual images. To address these issues, this paper proposes a radar-camera robust fusion network (RCRFNet), which leverages self-supervised learning and open-set recognition to effectively utilise the complementary information from both sensors. Specifically, the network uses matched radar-camera data through a frustum association approach to generate self-supervised signals, enhancing network training. The integration of global and local depth consistencies between radar point clouds and visual images, along with image features, helps construct object class confidence levels for detecting unknown targets. Additionally, these techniques are combined with a multi-layer feature extraction backbone and a multimodal feature detection head to achieve robust object detection. Experiments on the nuScenes public dataset demonstrate that RCRFNet outperforms state-of-the-art (SOTA) methods, particularly in conditions of low visual visibility and when detecting unknown class objects.

CRLNet: A Multimodal Peach Detection Network Based on Cooperative Asymptotic Enhancement and the Fusion of Granularity Refinement.

Accurate peach detection is essential for automated agronomic management, such as mechanical peach harvesting. However, ubiquitous occlusion makes identifying peaches from complex backgrounds extremely challenging. In addition, it is difficult to capture fine-grained peach features from a single RGB image, which can suffer from light and noise in scenarios with dense small target clusters and extreme light. To solve these problems, this study proposes a multimodal detector, called CRLNet, based on RGB and depth images. First, YOLOv9 was extended to design a backbone network that can extract RGB and depth features in parallel from an image. Second, to address the problem of information fusion bias, the Rough-Fine Hybrid Attention Fusion Module (RFAM) was designed to combine the advantageous information of different modes while suppressing the hollow noise at the edge of the peach. Finally, a Transformer-based Local-Global Joint Enhancement Module (LGEM) was developed to jointly enhance the local and global features of peaches using information from different modalities in order to enhance the percentage of information about the target peaches and remove the interference of redundant background information. CRLNet was trained on the Peach dataset and evaluated against other state-of-the-art methods; the model achieved an mAP50 of 97.1%. In addition, CRLNet also achieved an mAP50 of 92.4% in generalized experiments, validating its strong generalization capability. These results provide valuable insights for peach and other outdoor fruit multimodal detection.

Multimodal Hateful Meme Classification Based on Transfer Learning and a Cross-Mask Mechanism

Hateful memes are malicious and biased sentiment information widely spread on the internet. Detecting hateful memes differs from traditional multimodal tasks because, in conventional tasks, visual and textual information align semantically. However, the challenge in detecting hateful memes lies in their unique multimodal nature, where images and text in memes may be weak or unrelated, requiring models to understand the content and perform multimodal reasoning. To address this issue, we introduce a multimodal fine-grained hateful memes detection model named “TCAM”. The model leverages advanced encoding techniques from TweetEval and CLIP and introduces enhanced Cross-Attention and Cross-Mask Mechanisms (CAM) in the feature fusion stage to improve multimodal correlations. It effectively embeds fine-grained features of data and image descriptions into the model through transfer learning. This paper uses the Area Under the Receiver Operating Characteristic Curve (AUROC) as the primary metric to evaluate the model’s discriminatory ability. This approach achieved an AUROC score of 0.8362 and an accuracy score of 0.764 on the Facebook Hateful Memes Challenge (FHMC) dataset, confirming its high discriminatory capability. The TCAM model demonstrates relatively superior performance compared to ensemble machine learning methods.

Real-Time Multimodal 3D Object Detection with Transformers

The accuracy and real-time performance of 3D object detection are key factors limiting its widespread application. While cameras capture detailed color and texture features, they lack depth information compared to LiDAR. Multimodal detection combining both can improve results but incurs significant computational overhead, affecting real-time performance. To address these challenges, this paper presents a real-time multimodal fusion model called Fast Transfusion that combines the benefits of LiDAR and camera sensors and reduces the computational burden of their fusion. Specifically, our Fast Transfusion method uses QConv (Quick Convolution) to replace the convolutional backbones compared to other models. QConv concentrates the convolution operations at the feature map center, where the most information resides, to expedite inference. It also utilizes deformable convolution to better match the actual shapes of detected objects, enhancing accuracy. And the model incorporates EH Decoder (Efficient and Hybrid Decoder) which decouples multiscale fusion into intra-scale interaction and cross-scale fusion, efficiently decoding and integrating features extracted from multimodal data. Furthermore, our proposed semi-dynamic query selection refines the initialization of object queries. On the KITTI 3D object detection dataset, our proposed approach reduced the inference time by 36 ms and improved 3D AP by 1.81% compared to state-of-the-art methods.

Transfer Learning in Multimodal Sunflower Drought Stress Detection

Efficient water supply and timely detection of drought stress in crops to increase yields is an important task considering that agriculture is the primary consumer of water globally. This is particularly significant for plants such as sunflowers, which are an important source of quality edible oils, essential for human nutrition. Traditional detection methods are labor-intensive, time-consuming, and rely on advanced sensor technologies. We introduce an innovative approach based on neural networks and transfer learning for drought stress detection using a novel dataset including 209 non-invasive rhizotron images and 385 images of manually cleaned sections of sunflowers, subjected to normal watering or water stress. We used five neural network models: VGG16, VGG19, InceptionV3, DenseNet, and MobileNet, pre-trained on the ImageNet dataset, whose performance was compared to select the most efficient architecture. Accordingly, the most efficient model, MobileNet, was further refined using different data augmentation mechanisms. The introduction of targeted data augmentation and the use of grayscale images proved to be effective, demonstrating improved results, with an F1 score and an accuracy of 0.95. This approach encourages advances in water stress detection, highlighting the value of artificial intelligence in improving crop health monitoring and management for more resilient agricultural practices.

Multiple-readout lateral flow immunoassay for the sensitive detection of nitrofurazone metabolites through ultrabright AIE-MOF coupled in-situ growth strategy

The multiple-readout capability of multimodal detection enhances the flexibility, reliability, and accuracy of lateral flow immunoassay (LFIA). The conjugation of two different metal-organic frameworks (MOFs) as a new-generation composite material offers extraordinary opportunities for developing multimodal LFIA. It is anticipated to compensate limitations of traditional single colorimetric signal LFIA and improve the analysis performance. Herein, an ultra-bright fluorescent AIE-MOF was proposed and coupled with an in-situ growth of Prussian blue (PB) nanoparticles strategy to obtain a novel multimodal signal tracer (AIE-MOF@PB). Thereafter, it was successfully applied to develop the multimodal LFIA platform for the detection of nitrofurazone metabolites. The synergy of AIE-MOF and PB endows AIE-MOF@PB with superb water dispersibility, robust fluorescence emission, brilliant colorimetric signal, marvelous photothermal conversion, and enhanced antibody coupling efficiency, all of which facilitate a highly sensitive triple-readout LFIA platform. The detection sensitivity improved by at least 5-fold compared with the colloidal gold-based LFIA. This work not only inspires the rational design of aggregation-induced emission luminogens (AIEgen)-based complex materials but also highlights the promising potential in flexible point-of-care applications.

Tomographic Reconstruction Of Combustion Flow Fields From Density Measurements Based On Physics-Informed Neural Networks

Achieving stable, low-emission combustion with green hydrogen is crucial for climate-neutral ground-based power generation in turbomachinery. Lean combustion modes with green hydrogen reduce fuel consumption but increase unsteadiness. Thus, multimodal detection techniques for parameters like density and flow velocity are essential to understand the interconnected behavior of combustion, advection velocity and noise production. We previously introduced a high-speed camera-based laser interferometric vibrometer system to detect thermoacoustic oscillations and record advection velocities, using interferometric detection of density fluctuations and correlation-based velocity estimation. However, this method only estimates integral velocity fields, necessitating the solution of the inverse problem. This is relying on iterative optimization, which is computationally expensive and struggles with high dimensional, noisy or limited data. Physics-Informed Neural Networks offer an innovative and efficient approach to these problems, combining neural network flexibility with physical law constraints to unravel intricate cause-effect relationships. Here an approach for reconstructing local velocity fields from a single projection velocity calculated from integral density data is presented. Using a U-net and model assumptions for coupling local and integral velocity fields, the training minimizes errors between measured integral input fields and the predicted local field integrals. The comparison of measured local velocity and the network prediction achieved a relative mean squared error of 3 %.

Multimodal Gesture Recognition with Spatio-Temporal Features Fusion Based on YOLOv5 and MediaPipe

As a natural, intuitive and easy-to-learn mode of interaction, gesture plays an important role in communication. Hand detection, containing multimodal information, includes static and dynamic detection and involves intricate spatial relationship problems such as different hand sizes, complex joints, occlusion and self-occlusion. This study focused on a multimodal hand gesture recognition system based on YOLOv5 and MediaPipe with fused spatio-temporal features. First, the Mediapipe and OpenCV libraries were employed to implement hand keypoint detection. Subsequently, the human–computer interaction (HCI) of volume control was realized by identifying the distance between thumb and index. Finally, model training was conducted based on the YOLOv5 algorithm, and the recognition of different gesture categories was realized. The performance was evaluated and compared through YOLOv5s, YOLOv5m, and YOLOv5l. The gesture recognition system interface visualization was achieved through pyqt5. Experiments show that the average detection accuracy of the model is 99.4% and the recognition speed is around 0.2[Formula: see text]s.

Towards multimodal sarcasm detection via label-aware graph contrastive learning with back-translation augmentation

Multimodal sarcasm detection, as a sentiment analysis task, has witnessed great strides owing to the rapid development of multimodal machine learning. However, existing graph-based studies mainly focus on capturing the atomic-aware relations between textual and visual graphs within individual instances, neglecting label-aware connections between different instances. To address this limitation, we propose a novel Label-aware Graph Contrastive Learning (LGCL) method that detects ironic cues from a label-aware perspective of multimodal data. We first construct unimodal graphs for each instance and fuse them into graph semantic space, to obtain the multimodal graphs. Then, we introduce two label-aware graph contrastive losses: Label-aware Unimodal Contrastive Loss (LUCL) and Label-aware Multimodal Contrastive Loss (LMCL), to make the model aware of the shared ironic cues related to sentiment labels within multimodal graph representations. Additionally, we propose Back-translation Data Augmentation (BTrA) for both textual and visual data to enhance contrastive learning, where different back-translation schemes are designed to generate a larger number of positive and negative samples. Experimental results on two public datasets demonstrate our method achieves state-of-the-art (SOTA) compared to previous methods.

Diagnostic value of multimodal cardiovascular imaging technology coupled with biomarker detection in elderly patients with coronary heart disease

Aims/Background Coronary heart disease is a common disease in the elderly and has a complex pathogenesis, which complicates the clinical diagnostic process. Thus, enhancing the diagnostic efficiency for coronary heart disease is imperative to improve the life expectancy of the elderly. This study aimed to explore the diagnostic value of multimodal cardiovascular imaging technology coupled with biomarker detection in elderly patients with coronary heart disease. Methods The medical records of 421 patients with suspected coronary heart disease obtained from the geriatric department of the First Affiliated Hospital of Hebei North University from February 2020 to February 2023 were retrospectively analysed. After excluding 10 patients who did not meet the inclusion criteria, the remaining 411 patients were included in this study. The included subjects had undergone coronary computed tomography angiography and were divided into coronary heart disease group (n=208) and non-coronary heart disease group (n=203) according to the diagnostic results. Multimodal cardiovascular imaging (coronary computed tomography angiography and echocardiography) and detection of serum biomarkers such as small dense low-density lipoprotein, lipoprotein a, and gamma-glutamyl transferase were performed in both groups. The clinical indicators of the two groups were compared, and the combined diagnostic efficacy of multimodal cardiovascular imaging and biomarker detection was evaluated. Results Compared to the non-coronary heart disease group, the coronary heart disease group had significantly higher levels of maximum area stenosis, total plaque volume, total plaque burden and fibrotic plaque volume (p < ..001), and lower left ventricular ejection fraction level (p < ..001). Additionally, the coronary heart disease group exhibited higher levels of left ventricular end-diastolic volume, left ventricular end-systolic volume and stroke volume than the non-coronary heart disease group (p < ..001), and had higher levels of small dense low-density lipoprotein, lipoprotein a and gamma-glutamyl transferase (p < ..001). Our results demonstrated that combined diagnosis had better diagnostic efficacy than individual approaches, marked by higher area under the curve and sensitivity of the former (p < ..001). Conclusion Multimodal cardiovascular imaging technology combined with biomarker detection can distinctly improve the accuracy of coronary heart disease diagnosis in elderly patients.

A novel multimodal depression diagnosis approach utilizing a new hybrid fusion method

In recent years, research has found that the impact of depression status primarily lies in patients’ language expression and facial expressions. Furthermore, facial expressions and intonation in speech exhibit a natural coexistence, making facial and vocal information core recognition indicators in depression identification. It is imperative to explore the effective use of deep learning methods for multimodal depression detection. We have proposed a novel trilateral bimodal encoding model (MEN), attentional decision fusion (ADF), and feature extraction fusion strategy. We employed a hybrid fusion approach that combines early intra-modality fusion with late inter-modality fusion, for multimodal depression diagnosis. In the feature extraction fusion component, we combine different representations of the same modality before inputting them into the network for training, enhancing features relevant to depression in the data. Through our multimodal encoding network, we extract frame-level information using Convolutional Neural Networks (CNN) while considering long-term context information and dependencies with Bidirectional Long Short-Term Memory (BiLSTM). Finally, the three streams of information were effectively integrated through attention fusion representation in our Attention Decision Fusion module (ADF), for depression score regression prediction. Extensive experiments were conducted on two public datasets, AVEC2013 and AVEC2014. The average absolute error/ root mean squared error (MAE/RMSE) scores for predicting depression scores were 6.48/8.91 and 7.01/9.38, respectively. This demonstrated that our hybrid fusion method outperforms traditional early or late fusion methods in terms of performance.

Multi-modality multi-label ocular abnormalities detection with transformer-based semantic dictionary learning.

Blindness is preventable by early detection of ocular abnormalities. Computer-aided diagnosis for ocular abnormalities is built by analyzing retinal imaging modalities, for instance, Color Fundus Photography (CFP). This research aims to propose a multi-label detection of 28 ocular abnormalities consisting of frequent and rare abnormalities from a single CFP by using transformer-based semantic dictionary learning. Rare labels are usually ignored because of a lack of features. We tackle this condition by adding the co-occurrence dependency factor to the model from the linguistic features of the labels. The model learns the relation between spatial features and linguistic features represented as a semantic dictionary. The proposed method treats the semantic dictionary as one of the main important parts of the model. It acts as the query while the spatial features are the key and value. The experiments are conducted on the RFMiD dataset. The results show that the proposed method achieves the top 30% in Evaluation Set on the RFMiD dataset challenge. It also shows that treating the semantic dictionary as one of the strong factors in model detection increases the performance when compared with the method that treats the semantic dictionary as a weak factor.

MMFG: Multimodal-based Mutual Feature Gating 3D Object Detection

To address the problem that image and point cloud features are fused in a coarse fusion way and cannot achieve deep fusion, this paper proposes a multimodal 3D object detection architecture based on a mutual feature gating mechanism. First, since the feature aggregation approach based on the set abstraction layer cannot obtain fine-grained features, a point-based self-attention mechanism module is designed. This module is added to the extraction branch of point cloud features to achieve fine-grained feature aggregation while maintaining accurate location information. Second, a new gating mechanism is designed for the deep fusion of image and point cloud. Deep fusion is achieved by mutual feature weighting between the image and the point cloud. The newly fused features are then fed into a feature refinement network to extract classification confidence and 3D target bounding boxes. Finally, a multi-scale detection architecture is proposed to obtain a more complete object shape. The location-based encoding feature algorithm is also designed to focus the interest points in the region of interest adaptively. The whole architecture shows outstanding performance on the KITTI3D and nuSenece datasets, especially at the difficult level. It shows that the framework solves the problem of low detection rates in LiDAR mode due to the low number of surface points obtained from distant objects.