Decision-level Fusion Strategy Research Articles

Enhancing Facial Expression Recognition through Light Field Cameras.

In this paper, we study facial expression recognition (FER) using three modalities obtained from a light field camera: sub-aperture (SA), depth map, and all-in-focus (AiF) images. Our objective is to construct a more comprehensive and effective FER system by investigating multimodal fusion strategies. For this purpose, we employ EfficientNetV2-S, pre-trained on AffectNet, as our primary convolutional neural network. This model, combined with a BiGRU, is used to process SA images. We evaluate various fusion techniques at both decision and feature levels to assess their effectiveness in enhancing FER accuracy. Our findings show that the model using SA images surpasses state-of-the-art performance, achieving 88.13% ± 7.42% accuracy under the subject-specific evaluation protocol and 91.88% ± 3.25% under the subject-independent evaluation protocol. These results highlight our model's potential in enhancing FER accuracy and robustness, outperforming existing methods. Furthermore, our multimodal fusion approach, integrating SA, AiF, and depth images, demonstrates substantial improvements over unimodal models. The decision-level fusion strategy, particularly using average weights, proved most effective, achieving 90.13% ± 4.95% accuracy under the subject-specific evaluation protocol and 93.33% ± 4.92% under the subject-independent evaluation protocol. This approach leverages the complementary strengths of each modality, resulting in a more comprehensive and accurate FER system.

UYF-Net: A fusion network for feasible domain recognition and obstacle detection based on infrared thermography

Achieving safe navigation of unmanned surface vehicle (USV) in low light conditions remains a focus of research. The key to achieving the intelligent navigation of USV is the precise segmentation of navigable areas and fast obstacle detection. There are several challenges in processing infrared thermal imaging shoreline images using only semantic segmentation networks, including difficulties in extracting shorelines due to blurred details at the edges of infrared images, segmenting small targets, and complex obstacles due to small temperature differences in complex backgrounds. To address these issues, this paper introduces a novel UYF-Net network. The proposed method combines the improved U-Net with the YOLOv5m using a decision-level fusion strategy to achieve a better speed-accuracy tradeoff. Experimental results demonstrate that our method outperforms all comparative approaches discussed in this paper, further confirming the enhanced precision in segmentation, increased detection speed, and more stable model training process of the proposed network.

Signed Fingermark Liveness Detection Method Based on Deep Residual Networks and Multimodal Decision Fusion

Fingermarks play an important role in document identification. At the same time, fingermarks on paper documents are often accompanied by signatures and background text, which introduce noise to the original fingermark textures and increase the difficulty of detection. A signed fingermark detection method based on deep residual networks and a decision-level fusion strategy was proposed to defend against spoofing attacks from fake fingermarks. Firstly, the multi-scale structure was introduced in the residual module, which improved the network’s depth and breadth without increasing the parameters. Then, the multi-probability label strategy was refined and employed to enhance the local encoding ability of the feature extraction. A score fusion strategy was designed, with weights allocated based on the difference in signed interference levels of local image blocks. Finally, a model fusion strategy based on evidence theory was suggested, which improved detection accuracy by leveraging complementarity between models. A large-scale fingermark database was established, which included real fingermarks made from real fingers and fake fingermarks made from various materials, and this was divided into two sub databases: signed and unsigned. The experimental results show that the proposed method achieves 96.16% accuracy based on the fingerprint dataset of the global liveness detection competition called LivDet2017 and achieves 99.30% accuracy based on the signed fingermark database, while it has good resistance to spoofing attacks from unknown materials.

Multi-modal sarcasm detection based on Multi-Channel Enhanced Fusion model

The voluminous quantity of data accessible on social media platforms offers insight into the sentiment disposition of individual users, where multi-modal sarcasm detection is often confounding. Existing sarcasm detection methods use different information fusion methods to combine information from different modalities but ignore hidden information within modalities and inconsistent information between modalities. Discovering the implicit information within the modalities and strengthening the information interaction between modalities is still an important challenge. In this paper, we propose a Multi-Channel Enhanced Fusion (MCEF) model for cross-modal sarcasm detection to maximize the information extraction between different modalities. Specifically, text extracted from images acts as a new modality in the front-end fusion models to augment the utilization of image semantic information. Then, we propose a novel bipolar semantic attention mechanism to uncover the inconsistencies among different modal features. Furthermore, a decision-level fusion strategy from a new perspective is devised based on four models to achieve multi-channel fusion, each with a distinct focus, to leverage their advantages and mitigate the limitations. Extensive experiments demonstrate that our model surpasses current state-of-the-art models in multi-modal sarcasm detection.

A Multimodal Late Fusion Framework for Physiological Sensor and Audio-Signal-Based Stress Detection: An Experimental Study and Public Dataset

Stress can be considered a mental/physiological reaction in conditions of high discomfort and challenging situations. The levels of stress can be reflected in both the physiological responses and speech signals of a person. Therefore the study of the fusion of the two modalities is of great interest. For this cause, public datasets are necessary so that the different proposed solutions can be comparable. In this work, a publicly available multimodal dataset for stress detection is introduced, including physiological signals and speech cues data. The physiological signals include electrocardiograph (ECG), respiration (RSP), and inertial measurement unit (IMU) sensors equipped in a smart vest. A data collection protocol was introduced to receive physiological and audio data based on alterations between well-known stressors and relaxation moments. Five subjects participated in the data collection, where both their physiological and audio signals were recorded by utilizing the developed smart vest and audio recording application. In addition, an analysis of the data and a decision-level fusion scheme is proposed. The analysis of physiological signals includes a massive feature extraction along with various fusion and feature selection methods. The audio analysis comprises a state-of-the-art feature extraction fed to a classifier to predict stress levels. Results from the analysis of audio and physiological signals are fused at a decision level for the final stress level detection, utilizing a machine learning algorithm. The whole framework was also tested in a real-life pilot scenario of disaster management, where users were acting as first responders while their stress was monitored in real time.

ReFuSeAct: Representation fusion using self-supervised learning for activity recognition in next generation networks

Over the years, wearable sensors have gained a lot of attention from the research community due to their non-invasive nature, adoption of sensors by general public, and their applicability in healthcare services. With the advancements in communication networks, machine learning methods, and wearable sensor deployment, it is essential to design a method that could accurately classify human activities while reducing the dependence on annotated data. Traditional machine learning approaches require large-scale annotated data in order to provide a reasonable recognition performance. Recently, self-supervised learning methods are proposed but they are either limited to single sensor devices or fail to model intra-modal correlations within the self-supervised learning paradigm. In this work, we propose Representation Fusion using Self-supervised learning for Activity Recognition (ReFuSeAct) framework that uses modality-specific encoders, attention encoders, and decision-level fusion strategies to address the aforementioned limitations. The self-supervised learning paradigm ensures that the method achieves better performance even with less amount of annotated data. The architecture proposed for modality-specific encoder ensures that extraction of representative features that could help in improving recognition performance. The feature-level fusion performed using the proposed attention encoders enhances the quality of representative features that could be used in supervised learning phase. Finally, the decision-level fusion strategy enhances the activity recognition accuracy in comparison to the single deep learning classifier. Our experimental analysis shows that the proposed approach records 9.1% improvement over semi-supervised learning baselines and more than 2% improvement in comparison to existing self-supervised learning approaches.

Get Your Foes Fooled: Proximal Gradient Split Learning for Defense Against Model Inversion Attacks on IoMT Data

The past decade has seen a rapid adoption of Artificial Intelligence (AI), specifically the deep learning networks, in Internet of Medical Things (IoMT) ecosystem. However, it has been shown recently that the deep learning networks can be exploited by adversarial attacks that not only make IoMT vulnerable to the data theft but also to the manipulation of medical diagnosis. The existing studies consider adding noise to the raw IoMT data or model parameters which not only reduces the overall performance concerning medical inferences but also is ineffective to the likes of deep leakage from gradients method. In this work, we propose proximal gradient split learning (PSGL) method for defense against the model inversion attacks. The proposed method intentionally attacks the IoMT data when undergoing the deep neural network training process at client side. We propose the use of proximal gradient method to recover gradient maps and a decision-level fusion strategy to improve the recognition performance. Extensive analysis show that the PGSL not only provides effective defense mechanism against the model inversion attacks but also helps in improving the recognition performance on publicly available datasets. We report 14.0$\%$, 17.9$\%$, and 36.9$\%$ gains in accuracy over reconstructed and adversarial attacked images, respectively.

Quantitative Analysis of Forest Water COD Value Based on UV–vis and FLU Spectral Information Fusion

As an important ecosystem on the earth, forests not only provide habitat and food for organisms but also play an important role in regulating environmental elements such as water, atmosphere, and soil. The quality of forest waters directly affects the health and stability of aquatic ecosystems. Chemical oxygen demand (COD) is commonly used to assess the concentration of organic matter and the pollution status of water bodies, which is helpful in assessing the impact of human activities on forest ecosystems. To effectively measure the COD value, water samples were prepared from Purple Mountain in Nanjing and nearby rivers and lakes. Using ultraviolet–visible (UV–vis) and fluorescence (FLU) spectroscopy combined with data fusion, the COD values of the forest water were accurately measured. Due to the large dimensionality of spectral data, the successive projections algorithm (SPA) and competitive adaptive reweighted sampling (CARS) were applied to the selection of characteristic wavelengths. By establishing a discriminant model for single-level data and using the voting mechanism to fuse the output results of different models, a relatively high determination coefficient (R2) of 0.9932 and a low root-mean-square error (RMSE) of 0.4582 were obtained based on the decision-level data fusion model. Compared with the single-spectrum and feature-level fusion models, the decision-level fusion scheme achieves an efficient, comprehensive, and accurate quantification of the water COD value. This study has important applications in forest protection, water resources management, sewage treatment, and the food processing field.

Exploring fusion strategies for accurate RGBT visual object tracking

We address the problem of multi-modal object tracking in video and explore various options available for fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities, including pixel-level, feature-level and decision-level fusion. Specifically, in contrast to the existing approaches, we propose and develop the paradigm for combining multi-modal information for image fusion at pixel level. At the feature level, two different kinds of fusion strategies are investigated for completeness, i.e., the attention-based online fusion strategy and the offline-trained fusion block. At the decision level, a novel fusion strategy is put forward, inspired by the success of the simple averaging configuration which has shown so much promise. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of the proposed decision fusion method produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). A comprehensive comparison of the innovative pixel and feature-level fusion strategies with the proposed decision-level fusion method highlights the advantages fusing multimodal information at the decision score level. Extensive experimental results on five challenging datasets, i.e., GTOT, VOT-RGBT2019, RGBT234, LasHeR and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. The Code is available at https://github.com/Zhangyong-Tang/DFAT.

Credible Remote Sensing Scene Classification Using Evidential Fusion on Aerial-Ground Dual-View Images

Due to their ability to offer more comprehensive information than data from a single view, multi-view (e.g., multi-source, multi-modal, multi-perspective) data are being used more frequently in remote sensing tasks. However, as the number of views grows, the issue of data quality is becoming more apparent, limiting the potential benefits of multi-view data. Although recent deep neural network (DNN)-based models can learn the weight of data adaptively, a lack of research on explicitly quantifying the data quality of each view when fusing them renders these models inexplicable, performing unsatisfactorily and inflexibly in downstream remote sensing tasks. To fill this gap, in this paper, evidential deep learning is introduced to the task of aerial-ground dual-view remote sensing scene classification to model the credibility of each view. Specifically, the theory of evidence is used to calculate an uncertainty value which describes the decision-making risk of each view. Based on this uncertainty, a novel decision-level fusion strategy is proposed to ensure that the view with lower risk obtains more weight, making the classification more credible. On two well-known, publicly available datasets of aerial-ground dual-view remote sensing images, the proposed approach achieves state-of-the-art results, demonstrating its effectiveness.

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning.

Recently, personality trait recognition, which aims to identify people's first impression behavior data and analyze people's psychological characteristics, has been an interesting and active topic in psychology, affective neuroscience and artificial intelligence. To effectively take advantage of spatio-temporal cues in audio-visual modalities, this paper proposes a new method of multimodal personality trait recognition integrating audio-visual modalities based on a hybrid deep learning framework, which is comprised of convolutional neural networks (CNN), bi-directional long short-term memory network (Bi-LSTM), and the Transformer network. In particular, a pre-trained deep audio CNN model is used to learn high-level segment-level audio features. A pre-trained deep face CNN model is leveraged to separately learn high-level frame-level global scene features and local face features from each frame in dynamic video sequences. Then, these extracted deep audio-visual features are fed into a Bi-LSTM and a Transformer network to individually capture long-term temporal dependency, thereby producing the final global audio and visual features for downstream tasks. Finally, a linear regression method is employed to conduct the single audio-based and visual-based personality trait recognition tasks, followed by a decision-level fusion strategy used for producing the final Big-Five personality scores and interview scores. Experimental results on the public ChaLearn First Impression-V2 personality dataset show the effectiveness of our method, outperforming other used methods.

Suspicious Object Detection for Millimeter-Wave Images with Multi-View Fusion Siamese Network.

Millimeter-wave (MMW) imaging techniques have been widely used in the public security industries for their under-controlled privacy concerns and no health hazards. However, since MMW images are low resolution and most objects are small, reflection-weak, diverse, suspicious object detection in the MMW images is a very challenging task. This paper develops a robust suspicious object detector for the MMW images based on the Siamese network integrated with the pose estimation and image segmentation, which estimates the coordinates of human joints and segments the complete human images into symmetrical body part images. Unlike most existing detectors, which detect and recognize suspicious objects in MMW images and require a complete training set with correct annotations, our proposed model aims to learn the similarity between two symmetrical human body part images segmented from the complete MMW images. Furthermore, to decrease the misdetection caused by the restricted field of view, we further fuse the multi-view MMW images observed from the same person by designing a decision-level fusion strategy and feature-level fusion strategy based on the attention mechanism. Experimental results on the measured MMW images show that our proposed models have favorable detection accuracy and speed in practical application and thus prove their effectiveness.

Quantitative analysis of Chinese steamed bread staling using NIR, MIR, and Raman spectral data fusion

For efficient and comprehensive detection of the staling degree of Chinese steamed bread (CSB), staled CSB samples stored for 0–16 days were prepared and analyzed using near-infrared (NIR), mid-infrared (MIR), and Raman spectroscopy combined with data fusion. Among three data fusion schemes, decision-level fusion achieved the best performance when quantifying the CSB staling degree according to the soluble starch amylose fraction, relative crystallinity, and hardness, with determination coefficients and root mean square errors for the validation set in the range of 0.928–0.986 and 0.015–1.290, respectively. The relative percent deviation values of the three indicators increased to 8.362, 4.735, and 3.617, respectively. These results indicate that the combination of NIR, MIR, and Raman spectroscopy as a decision-level fusion scheme can achieve efficient, comprehensive, and accurate quantification of the staling degree of CSB. This research has important applications for food quality, safety, and shelf-life evaluations.

Medical College Education Data Analysis Method Based on Improved Deep Learning Algorithm

Deep learning (DL) has become a popular study topic in the field of artificial intelligence (AI) in recent years, due to its significant role in various application areas. It leverages supercomputing capacity in the era of big data to uncover the high-level abstract ideas in the original dataset and serves as decision support in the application sector by increasing the number of channels and the scale of parameters. This study designs and implements a heterogeneous medical education data analysis system based on DL technology. The proposed system adopts DL technology to model, analyzes the heterogeneous medical education data, uses the decision-level fusion strategy for the data model, and designs and implements the voting method and the weighting method. The decision value is statistically calculated to realize the improved DL algorithm for the medical college education data analysis method. In addition, this study also uses the Alzheimer’s disease public dataset with various structures and modalities of medical education data to compare and evaluate the systematic data preprocessing model performance and the effect of fusion methods. The experimental result validates the proposed model’s performance, demonstrating that the way of evaluating complete heterogeneous multimodal data is not only closer to the genuine diagnostic process but also aids clinicians in grasping the patient’s entire state and obtaining outcomes. Further, the essential ideas and implementation techniques of convolutional neural network (CNN) and stacked autoencoder as well as its application cases in medical college education data analysis are thoroughly explained.

Structural damage detection based on decision-level fusion with multi-vibration signals

When a structure is damaged, its vibration signals change. If a single vibration signal is used for structural damage detection (SDD), it may sometimes lead to low detection accuracy. To avoid this phenomenon, this paper presents a SDD method based on decision-level fusion (DLF) with multi-vibration signals. In this study, acceleration (ACC), strain (E), displacement (DIS), and the fusion signal of all three of these signals (ACC, E and DIS), are studied. The damage information can be extracted from the vibration signal of a structure by using convolution neural networks (CNN). The above four vibration signals are used as the inputs to train four CNN models, and each model outputs a corresponding result. Finally, a DLF strategy is used to fuse the detection results of each CNN. To demonstrate the effectiveness and correctness of the proposed method, a steel frame bridge is investigated with numerical simulations and vibration experiments. The research shows that the damage detection method based on DLF with multi-vibration signals can effectively improve the accuracy of the CNN damage detection.

Learning inter-class optical flow difference using generative adversarial networks for facial expression recognition

Facial expression recognition is a fine-grained task because different emotions have subtle facial movements. This paper proposes to learn inter-class optical flow difference using generative adversarial networks (GANs) for facial expression recognition. Initially, the proposed method employs a GAN to produce inter-class optical flow images from the difference between the static fully expressive samples and neutral expression samples. Such inter-class optical flow difference is used to highlight the displacement of facial parts between the neutral facial images and fully expressive facial images, which can avoid the disadvantage that the optical flow change between adjacent frames of the same video expression image is not obvious. Then, the proposed method designs four-channel convolutional neural networks (CNNs) to learn high-level optical flow features from the produced inter-class optical flow images, and high-level static appearance features from the fully expressive facial images, respectively. Finally, a decision-level fusion strategy is adopted to implement facial expression classification. The proposed method is validated on two public facial expression databases, BAUM_1a, SAMM and AFEW5.0, demonstrating its promising performance.

Score, Rank, and Decision-Level Fusion Strategies of Multicode Electromyogram-Based Verification and Identification Biometrics.

Recent advances in biometric research have established surface electromyogram (sEMG) as a potential spoof-free solution to address some key limitations in current biometric traits. The nature of sEMG signals provide a unique dual-mode security: sEMGs have individual-specific characteristics (biometrics), and users can customize and change gestures just like passcodes. Such security also facilitates the use of code sequences (multicode) to further enhance the security. In this study, three levels of fusion, score, rank, and decision were investigated for two biometric applications, verification and identification. This study involved 24 subjects performing 16 hand/finger gestures, and code sequences with varying codelengths were generated. The performance of the verification and identification system was analyzed for varying codelength (M: 1-6) and rank (K: 1-4) to determine the best fusion scheme and desirable parameter values for a multicode sEMG biometric system. The results showed that the decision-level fusion scheme using a weighted majority voting resulted in an average equal error rate of 0.6% for the verification system when M = 4. For the identification system, the score-level fusion scheme with score normalization based on fitting a Weibull distribution resulted in a minimum false rejection rate of 0.01% and false acceptance rate of 4.7% using a combination of K = 2 and M = 4. The results also suggested that the parameters M and K could be adjusted based on the number of users in the database to facilitate optimal performance. In summary, a multicode sEMG biometric system was developed to provide improved dual-mode security based on the personalized codes and biometric traits of individuals, with the combination of enhanced security and flexibility.

MDMN: Multi-task and Domain Adaptation based Multi-modal Network for early rumor detection

With the development of social media, people tend to express their opinions on multimedia such as text, photos, audios, and videos. Meanwhile, more rumors hiding in multi-modal content are misleading social media users. The early rumor detection task aims to detect rumors before spreading. However, annotation on multi-modal data often involves a large amount of manpower. Existing approaches universally used transfer learning to overcome it. But they ignored the differences between the source domain of pre-trained models and the task domain. In this paper, Multi-task and Domain Adaptation based Multi-modal Network (MDMN) is proposed, which consists of three components: Textual Feature Extractor, Visual Feature Extractor, and Fusion & Classification Network. To improve the diversity and stability of textual representation, a Multi-task Sharing Layer, a task-specific Transformer Encoder and a Selection Layer are applied. Domain Adaptation is involved in training an adaptive model for extracting visual representation. The adaptive models can encode task data better than fine-tuning the pre-trained models. Then, multi-modal representations are fused through two fusion strategies, each having their own benefits. The experiment on multi-modal datasets collected from Weibo and Twitter show that the proposed MDMN can outperform the baseline methods. The decision-level fusion strategy achieves a Recall of over 92%.

A complementary and precise vehicle detection approach in RGB-T images via semi-supervised transfer learning and decision-level fusion

ABSTRACT Owing to the strong complementarity between visible and infra-red thermal (RGB-T) images, the two types of images receive the increased application of vehicle detection in traffic monitoring. Now, how to take full advantage of the two kinds of images for vehicle fusion detection has drawn wide publicity. Nevertheless, due to the requirement of high quality and high precision on vehicle fusion detection, and insufficiency of the infra-red thermal dataset, it is extremely hard to realize the vehicle fusion detection in RGB-T images. In our work, a complementary and precise vehicle fusion detection approach in RGB-T images is put forward, which combines the detection results of both visible and infra-red thermal images based on a decision-level fusion strategy. For vehicle detection in visible images, an effective and shallow network is constructed. For vehicle detection in infra-red thermal images, we do not directly train an effective network because of the insufficiency of the infra-red thermal vehicle samples. Instead, semi-supervised transfer learning is applied to combine the trained visible network for vehicle detection in infra-red thermal images. Next, for the decision-level fusion, we propose a novel fusion algorithm. It not only retains the vehicle results detected only in the visible images or only the infra-red thermal images but also fuses the detection results of the same vehicle in both visible and infra-red thermal images. Finally, VOT2019 and RGBT234 data sets are introduced to evaluate the proposed vehicle fusion detection approach. Compared with the main trend of current, it is indicated that the proposed approach can obtain competitive and superior vehicle fusion detection results in RGB-T images.

Synergy of Sentinel-1 and Sentinel-2 Imagery for Early Seasonal Agricultural Crop Mapping

The exploitation of the unprecedented capacity of Sentinel-1 (S1) and Sentinel-2 (S2) data offers new opportunities for crop mapping. In the framework of the SenSAgri project, this work studies the synergy of very high-resolution Sentinel time series to produce accurate early seasonal binary cropland mask and crop type map products. A crop classification processing chain is proposed to address the following: (1) high dimensionality challenges arising from the explosive growth in available satellite observations and (2) the scarcity of training data. The two-fold methodology is based on an S1-S2 classification system combining the so-called soft output predictions of two individually trained classifiers. The performances of the SenSAgri processing chain were assessed over three European test sites characterized by different agricultural systems. A large number of highly diverse and independent data sets were used for validation experiments. The agreement between independent classification algorithms of the Sentinel data was confirmed through different experiments. The presented results assess the interest of decision-level fusion strategies, such as the product of experts. Accurate crop map products were obtained over different countries in the early season with limited training data. The results highlight the benefit of fusion for early crop mapping and the interest of detecting cropland areas before the identification of crop types.