Feature Level Fusion Technique Research Articles

Comparison of data fusion strategies for automated prostate lesion detection using mpMRI correlated with whole mount histology

BackgroundIn this work, we compare input level, feature level and decision level data fusion techniques for automatic detection of clinically significant prostate lesions (csPCa).MethodsMultiple deep learning CNN architectures were developed using the Unet as the baseline. The CNNs use both multiparametric MRI images (T2W, ADC, and High b-value) and quantitative clinical data (prostate specific antigen (PSA), PSA density (PSAD), prostate gland volume & gross tumor volume (GTV)), and only mp-MRI images (n = 118), as input. In addition, co-registered ground truth data from whole mount histopathology images (n = 22) were used as a test set for evaluation.ResultsThe CNNs achieved for early/intermediate / late level fusion a precision of 0.41/0.51/0.61, recall value of 0.18/0.22/0.25, an average precision of 0.13 / 0.19 / 0.27, and F scores of 0.55/0.67/ 0.76. Dice Sorensen Coefficient (DSC) was used to evaluate the influence of combining mpMRI with parametric clinical data for the detection of csPCa. We compared the DSC between the predictions of CNN’s trained with mpMRI and parametric clinical and the CNN’s trained with only mpMRI images as input with the ground truth. We obtained a DSC of data 0.30/0.34/0.36 and 0.26/0.33/0.34 respectively. Additionally, we evaluated the influence of each mpMRI input channel for the task of csPCa detection and obtained a DSC of 0.14 / 0.25 / 0.28.ConclusionThe results show that the decision level fusion network performs better for the task of prostate lesion detection. Combining mpMRI data with quantitative clinical data does not show significant differences between these networks (p = 0.26/0.62/0.85). The results show that CNNs trained with all mpMRI data outperform CNNs with less input channels which is consistent with current clinical protocols where the same input is used for PI-RADS lesion scoring.Trial registrationThe trial was registered retrospectively at the German Register for Clinical Studies (DRKS) under proposal number Nr. 476/14 & 476/19.

Multi-dimensional CNN Based Feature Extraction with Feature Fusion and SVM for Human Activity Recognition in Surveillance Videos

Background/Objectives: The accurate recognition of human activities from video sequences is very challenging due to low resolution, cluttered background, partial occlusion, and different viewpoints. Machine learning (ML) based automated HAR from surveillance videos is required with the fusion of various feature extraction techniques. Methods: In this paper, SVM with feature fusion is utilized for automatic recognition from surveillance videos. A Histogram of Oriented Gradient (HOG) is used to segment the frame to differentiate humans from other objects or background noise in the input video frames. The multi-feature extraction can be accomplished in terms of Gabor Wavelet Transform (GWT), Autocorrelogram, Gray-Level Co-Occurrence Matrix (GLCM), HSV histogram, and Multi-dimensional CNN. The proposed approach is implemented in MATLAB software and compared with existing approaches like Space-Time Interest Point (STIP) and Histogram of Optical Flow (HOF). Findings: The proposed approach outperforms the existing approaches in terms of reduced time consumption and high accuracy, 99.886% when using the UCF101 dataset and 99.538% when using the UTKinect dataset. Novelty: The most discriminative feature information is obtained with the feature-level fusion technique. From the feature information, various human actions are recognized with the classification algorithm. Keywords: Human activity recognition, Machine Learning, Surveillance Videos, Human detection algorithm, Feature extraction, SVM classifier

Research on Wind Speed Prediction Method Based on Feature Fusion

Wind speed occupies a central position in the study of atmospheric circulation and climate change, playing a crucial role in the accuracy and reliability of meteorological forecasting models. The current meteorological observation network, especially in remote and mountainous areas, faces challenges due to insufficient data resulting from low station density. This limitation hampers a comprehensive understanding of wind field dynamics, subsequently affecting the performance of forecasting models. To enhance the performance of meteorological prediction models, this study adopts an innovative approach that combines boxplot difference indexing and feature-level fusion techniques to filter key predictive features and reduce information redundancy. Furthermore, by integrating spatial location information, the contribution of geographic data to wind field prediction is enhanced, improving the quality of the fused data and thereby increasing the accuracy of model predictions. Experimental comparisons demonstrate that the integrated wind field dataset, post feature fusion, achieves better results across various deep learning models.

An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network

Sentiment analysis is a key component of many social media analysis projects. Additionally, prior research has concentrated on a single modality in particular, such as text descriptions for visual information. In contrast to standard image databases, social images frequently connect to one another, making sentiment analysis challenging. The majority of methods now in use consider different images individually, rendering them useless for interrelated images. We proposed a hybrid Arithmetic Optimization Algorithm- Hunger Games Search (AOA-HGS)-optimized Ensemble Multi-scale Residual Attention Network (EMRA-Net) technique in this paper to explore the modal correlations including texts, audio, social links, and video for more effective multimodal sentiment analysis. The hybrid AOA-HGS technique learns complementary and comprehensive features. The EMRA-Net uses two segments, including Ensemble Attention CNN (EA-CNN) and Three-scale Residual Attention Convolutional Neural Network (TRA-CNN), to analyze the multimodal sentiments. The loss of spatial domain image texture features can be reduced by adding the Wavelet transform to TRA-CNN. The feature-level fusion technique known as EA-CNN is used to combine visual, audio, and textual information. The proposed method performs significantly better than the existing multimodel sentimental analysis techniques of HALCB, HDF, and MMLatch when evaluated using the Multimodal Emotion Lines Dataset (MELD) and EmoryNLP datasets. Also, even though the size of the training set varies, the proposed method outperformed other techniques in terms of recall, accuracy, F score, and precision and takes less time to compute in both datasets.

Hybridization of Deep Sequential Network for Emotion Recognition Using Unconstraint Video Analysis

The reliable way to discern human emotions in various circumstances has been proven to be through facial expressions. Facial expression recognition (FER) has emerged as a research topic to identify various essential emotions in the present exponential rise in research for emotion detection. Happiness is one of these basic emotions everyone may experience, and facial expressions are better at detecting it than other emotion-measuring methods. Most techniques have been designed to recognize various emotions to achieve the highest level of general precision. Maximizing the recognition accuracy for a particular emotion is challenging for researchers. Some techniques exist to identify a single happy mood recorded in unrestricted video. Still, they are all limited by the processing of extreme head posture fluctuations that they need to consider, and their accuracy still needs to be improved. This research proposes a novel hybrid facial emotion recognition using unconstraint video to improve accuracy. Here, a Deep Belief Network (DBN) with long short-term memory (LSTM) is employed to extract dynamic data from the video frames. The experiments conducted uses decision-level and feature-level fusion techniques are applied unconstrained video dataset. The outcomes show that the proposed hybrid approach may be more precise than some existing facial expression models.

CMOT: A cross-modality transformer for RGB-D fusion in person re-identification with online learning capabilities

Person re-identification (reID) is a crucial aspect of intelligent surveillance systems, enabling the recognition of individuals across non-overlapping camera views. As compared to the RGB modality, RGB-D based reID has the potential to achieve robust and high performance by leveraging rich complementary features of both modalities, making it applicable in various occluded scenarios. However, current multimodal reID approaches often rely on late fusion or feature-level fusion techniques to combine multiple modalities, which limits their ability to proficiently exploit complementary visual and depth-related semantic information and capture complex interactions between unimodal features. To address these limitations, this paper introduces a cross-modality online transformer (CMOT) for RGB-D based person reID with online learning capabilities, which effectively utilizes both RGB and depth modalities for the extraction of spatio-temporal features and fuses across modalities. Our CMOT is composed of three main components: (1) a hypothesis generation module based on a person detector and tracker, (2) dual-stream feature extractors via convolutional neural networks (CNNs), (3) and a fusion transformer based on a self-attention-driven self-attentive modality refinement module (SAMR) and a cross-attention-driven cross-attentive modality interaction module (CAMI) to refine and fuse RGB-D complementary features extracted from the dual-stream, RGB and depth stream, CNNs. Additionally, we introduce a bottleneck enhancement feed-forward block to enhance the model’s representation capability within SAMR and CAMI, significantly reducing parameters and computations compared to the traditional feed-forward network. Moreover, we design the triplet loss function with distance measuring ability for incorporating online learning and finally CMOT works as a few-shot network for reID. Experimental results on three RGB-D person re-identification datasets, namely BIWI RGBD-ID, RobotPKU RGBD-ID, and TVPR2, demonstrate the effectiveness and robustness of CMOT.

Orientation-Independent Human Activity Recognition Using Complementary Radio Frequency Sensing.

RF sensing offers an unobtrusive, user-friendly, and privacy-preserving method for detecting accidental falls and recognizing human activities. Contemporary RF-based HAR systems generally employ a single monostatic radar to recognize human activities. However, a single monostatic radar cannot detect the motion of a target, e.g., a moving person, orthogonal to the boresight axis of the radar. Owing to this inherent physical limitation, a single monostatic radar fails to efficiently recognize orientation-independent human activities. In this work, we present a complementary RF sensing approach that overcomes the limitation of existing single monostatic radar-based HAR systems to robustly recognize orientation-independent human activities and falls. Our approach used a distributed mmWave MIMO radar system that was set up as two separate monostatic radars placed orthogonal to each other in an indoor environment. These two radars illuminated the moving person from two different aspect angles and consequently produced two time-variant micro-Doppler signatures. We first computed the mean Doppler shifts (MDSs) from the micro-Doppler signatures and then extracted statistical and time- and frequency-domain features. We adopted feature-level fusion techniques to fuse the extracted features and a support vector machine to classify orientation-independent human activities. To evaluate our approach, we used an orientation-independent human activity dataset, which was collected from six volunteers. The dataset consisted of more than 1350 activity trials of five different activities that were performed in different orientations. The proposed complementary RF sensing approach achieved an overall classification accuracy ranging from 98.31 to 98.54%. It overcame the inherent limitations of a conventional single monostatic radar-based HAR and outperformed it by 6%.

Improved Feature-Level Fusion-Based Biometric System for Genuine and Imposter Identification

In this research work is divided into five sections: pre-processing, feature extraction, featurelevel fusion, imposter/ genuine classification, and male/female identification. The gathered raw data for palm print, dorsal vein, wrist vein, and palm vein are pre-processed with enhanced median filtering algorithms. The required features are then being extracted from the pre-processed images. Following that, an improved feature-level fusion technique is carried out based on the required security level. In the low security level, the certain image features are considered. Furthermore, if the medium security level is chosen, any two sets of attributes has been combined. If the high security level is chosen, all of the retrieved characteristics has been combined. Further, multiple classifications are carried out. It will determine whether the individual is genuine or an imposter. If it is an imposter, simply get rid of it. If the categorized result is genuine, it is reclassified as male or female.

An Efficient Epileptic Seizure Detection Technique using Discrete Wavelet Transform and Machine Learning Classifiers

This paper presents an epilepsy detection method based on discrete wavelet transform (DWT) with Machine learning classifiers. Here DWT has been used for feature extraction as it provides a better decomposition of the signals in different frequency bands. At first, DWT has been applied to the EEG signal to extract the detail and approximate coefficients or different sub-bands. After the extraction of the coefficients, Principal component analysis (PCA) has been applied on different sub-bands and then a feature level fusion technique is used to extract the main features in low dimensional feature space. Three classifiers name: Support Vector Machine (SVM) classifier, K-Nearest-Neighbor (KNN) classifier, and Naive Bayes (NB) classifier have been used in the proposed work for classifying the EEG signals. The raised method is tested over Bonn databases and provides a maximum of 100% recognition accuracy for KNN, SVM, NB classifiers.

Automated Teller Machine Authentication Using Biometric

This paper presents a novel method of a secured card-less Automated Teller Machine (ATM) authentication based on the three bio-metrics measures. It would help in the identification and authorization of individuals and would provide robust security enhancement. Moreover, it would assist in providing identification in ways that cannot be impersonated. To the best of our knowledge, this method of Biometric_ fusion way is the first ATM security algorithm that utilizes a fusion of three biometric features of an individual such as Fingerprint, Face, and Retina simultaneously for recognition and authentication. These biometric images have been collected as input data for each module in this system, like a fingerprint, a face, and a retina module. A database is created by converting these images to YIQ color space, which is helpful in normalizing the brightness levels of the image hence mainly (Y component’s) luminance. Then, it attempt to enhance Cellular Automata Segmentation has been carried out to segment the particular regions of interest from these database images. After obtaining segmentation results, the featured extraction method is carried out from these critical segments of biometric photos. The Enhanced Discrete Wavelet Transform technique (DWT Mexican Hat Wavelet) was used to extract the features. Fusion of extracted features of all three biometrics features have been used to bring in the multimodal classification approach to get fusion vectors. Once fusion vectors ware formulated, the feature level fusion technique is incorporated based on the extracted feature vectors. These features have been applied to the machine learning algorithm to identify and authorization of multimodal biometrics for ATM security. In the proposed approach, we attempt at useing an enhanced Deep Convolutional Neural Network (DCNN). A hybrid optimization algorithm has been selected based on the effectiveness of the features. The proposed approach results were compared with existing algorithms based on the classification accuracy to prove the effectiveness of our algorithm. Moreover, comparative results of the proposed method stand as a proof of more promising outcomes by combining the three biometric features.

Performance Analysis of Two-Stage Optimal Feature-Selection Techniques for Finger Knuckle Recognition

Automated biometric authentication attracts the attention of researchers to work on hand-based images to develop applications in forensics science. Finger Knuckle Print (FKP) is one of the hand-based biometrics used in the recognition of an individual. FKP is rich in texture, less in contact and known for its unique features. The dimensionality of the features, extracted from the image, is one of the main problems in pattern recognition. Since selecting the relevant features is an important but challenging task, the feature subset selection is an optimization problem. A reduced number of features results in enhanced classification accuracy. The proposed FKP system presents a mulitalgorithm fusion based on subspace algorithms at feature level fusion technique. In this paper, a new feature-selection algorithm, which is a Modified Magnetotatic bacterium Optimization Algorithm (MMBOA), is proposed for finger knuckle recognition to select relevant and useful features that increase the classification accuracy. The distinct characteristic of this bacterium influences the design of a new optimization technique. The hybrid features such as Eigen and Fisher (EiFi) are extracted from the finger knuckle. The fusion of this feature vector is optimized using newly proposed MMBOA_mr optimization algorithm. The results demonstrate a significant improvement compared with unimodal identifiers, and the proposed approach significantly outperforms with a recognition accuracy of 99.7% with 22 features with the reduction rate of 72%. Additionally, the proposed approach is compared with the state-of-the-art methods.

EMOTION ANALYSIS USING SIGNAL AND IMAGE PROCESSING APPROACH BY IMPLEMENTING DEEP NEURAL NETWORK

Emotion recognition is important in human communication and to achieve a complete interaction between humans and machines. In medical applications, emotion recognition is used to assist the children with Autism Spectrum Disorder (ASD to improve their socio-emotional communication, helps doctors with diagnosis of diseases such as depression and dementia and also helps the caretakers of older patients to monitor their well-being. This paper discusses the application of feature level fusion of speech and facial expressions of different emotions such as neutral, happy, sad, angry, surprise, fearful and disgust. Also, to explore how best to build the deep learning networks to classify the emotions independently and jointly from these two modalities. VGG-model is utilized to extract features from facial images, and spectral features are extracted from speech signals. Further, feature level fusion technique is adopted to fuse the features extracted from the two modalities. Principal Component Analysis (PCA is implemented to choose the significant features. The proposed method achieved a maximum score of 90% on training set and 82% on validation set. The recognition rate in case of multimodal data improved greatly when compared to unimodal system. The multimodal system gave an improvement of 9% compared to the performance of the system based on speech. Thus, result shows that the proposed Multimodal Emotion Recognition (MER outperform the unimodal emotion recognition system.

Geographical traceability and multielement analysis of edible and medicinal fungi: Taking Wolfiporia cocos (F.A. Wolf) Ryvarden and Gilb. as an example.

Different geographical environment has a certain influence on the accumulation of fungi elements and chemical components. However, our knowledge is limited to elucidate the fungi elements in response to heterogeneous environmental and the quality differences among different habitats. Here, multielement analysis, FTIR spectrum, and feature-level fusion technique combined with chemometrics were used to study Wolfiporia cocos from different geographical areas, different sampling sites and different altitude sources. From the results, (1) there is significant difference in element content of samples from different sampling sites and no positive correlation with geographical ranges. (2) There is a correlation between elevation and elements, and relatively low elevation (<1,800 m) is conducive to the enrichment of elements. (3) From the perspective of elements, the W. cocos in Yuxi have relatively better quality. (4) FTIR and feature-level models can well realize origin identification. The SVM models are better than the PLS-DA models, and the feature-level model is better than the single FTIR models. In summary, this study demonstrated that the developed method was reliable and could realize the genuineness evaluation and origin identification of W. cocos. The results have implications for the establishment of the technology system of geographical traceability and the development of high-quality geographical indication products of W. cocos.

Multimodal Biometric Recognition System Using Face and Finger Vein Biometric Traits with Feature and Decision Level Fusion Techniques

Multimodal Biometric Recognition System Using Face and Finger Vein Biometric Traits with Feature and Decision Level Fusion Techniques

Evaluation of different approaches to the fusion of Sentinel -1 SAR data and Resourcesat 2 LISS III optical data for use in crop classification

ABSTRACT This study evaluates various combinations of data fusion techniques at Pixel, Feature, and Decision level for crop classification using Sentinel-1 Synthetic Aperture Radar (SAR) data and Resourcesat-2 LISS (Linear Imaging Self Scanning) III, optical data for Yadgir District of Karnataka, India. For Pixel level data fusion, techniques such as brovey transformation (BT), principal component analysis (PCA), multiplicative transformation (MLT), and wavelet with IHS (intensity-hue-saturation) were used. Results were compared between different fusion techniques visually, statistically (using universal image quality index), and through image classification (Rule-based and Maximum likelihood) for major crops (Rice, Cotton, and Pigeon pea) in the area. The estimated crop area for all three major crops was compared with the Government statistics. Among the four pixel-level fusion techniques used, the wavelet method performed best in retaining the image quality. However, the study showed that using the feature-level fusion technique, maximum accuracy was obtained for Rice crop. In contrast, the decision-level fusion improved the efficiency for other crops (Cotton and Pigeon pea).

A Fusion of Palm Print and Palm Vein Features for Person Identification Using Grey Coding Technique

An effective bimodal palm print and palm vein biometric person recognition method is suggested in this paper. Palm print is referred to as the external biometric line comprising main lines, wrinkles, and ridges on the palm surface and the palm vein is the structure of the lines below the palm surface. For individuals, these patterns are unique, stable and they provide enormous important information for person identification. The features of both modalities are extracted using a Sobel operator and grey code encoding techniques. The suggested framework is rigorously checked on the CASIA multispectral database and uses decision level and feature level fusion techniques to achieve the best FAR 0.0047 & FRR 0.050. The Proposed image fusion technique results in performance improvement than the individual biometric identification.

Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis

Numerous public networks, namely Instagram, YouTube, Facebook, Twitter, etc., share their own feelings and idea as videotapes, posts, and pictures. In future research, adapting to such data and mining valuable information from it will be an undeniably troublesome errand. This paper proposes a novel audio–video–textual-based multimodal sentiment analysis approach. The proposed approach investigates the sentiments that are collected from the web recordings that utilize audio, video, and textual modalities for further extraction. A feature-level fusion technique is employed in fusing the extracted features from different modalities. Therefore, the extracted features are optimally chosen by using a novel oppositional grass bee optimization (OGBEE) algorithm to obtain the best optimal feature set. Here, 12 benchmark functions are developed to validate the numerical efficiency and the effectiveness of a novel OGBEE algorithm for various aspects. Moreover, our proposed approach utilizes multilayer perceptron-based neural network (MLP-NN) for sentiment classification. The experimental analysis reveals that the proposed approach provides better classification accuracy of about 95.2% with less computational time.

Multi-instance iris remote authentication using private multi-class perceptron on malicious cloud server

In recent years, biometric authentication system (BAS) has become the most promising and popular authentication system in identity management. Due to its capability to solve the limitations of unimodal systems, multi-biometric systems (MBS) have been extensively accepted in various fields. The main step in MBS is information fusion. On the other hand, directly storing the fused templates into a centralized server leads to privacy concerns. Recently, many BAS based on homomorphic encryption has been introduced to provide confidentiality for the fused templates. However, most of the existing solutions rely on an implication of the assumption that the server is “Honest-but-Curious”. As a result, the compromise of such server results into entire system vulnerability. To address this, we propose a novel P rivacy P reserving (PP) multi-instance iris remote authentication system to accord with attacks at the malicious server and over the transmission channel. Our scheme uses F ully H omomorphic E ncryption (FHE) to achieve the confidentiality of the fused iris templates and polynomial factorization algorithm to achieve the integrity of the matching result. We propose a PP iris authentication system using P rivate M ulti-C lass P erceptron (PMCP) by using the properties of FHE. Moreover, we propose C ontradistinguish S imilarity A nalysis (CSA), a feature level fusion technique that minimizes the between-class correlations and maximizes the pair-wise correlations. Our method has experimented on IITD and CASIA-V3-Interval iris databases to check the effectiveness and robustness. Experimental results show that our method provides improved accuracy, and eliminates the need to trust the cloud server when compared to the state-of-the-art approaches.

Feature-level fusion approaches based on multimodal EEG data for depression recognition

This study aimed to construct a novel multimodal model by fusing different electroencephalogram (EEG) data sources, which were under neutral, negative and positive audio stimulation, to discriminate between depressed patients and normal controls. The EEG data of different modalities were fused using a feature-level fusion technique to construct a depression recognition model. The EEG signals of 86 depressed patients and 92 normal controls were recorded simultaneously while receiving different audio stimuli. Then, from the EEG signals of each modality, linear and nonlinear features were extracted and selected to obtain features of each modality. In addition, a linear combination technique was used to fuse the EEG features of different modalities to build a global feature vector and find several powerful features. Furthermore, genetic algorithms were used to perform feature weighting to improve the overall performance of the recognition framework. The classification accuracy of each classifier, namely the k-nearest neighbor (KNN), decision tree (DT), and support vector machine (SVM), was compared, and the results were encouraging. The highest classification accuracy of 86.98% was obtained by the KNN classifier in the fusion of positive and negative audio stimuli, demonstrating that the fusion modality could achieve higher depression recognition accuracy rate compared with the individual modality schemes. This study may provide an additional tool for identifying depression patients.

CNN-BASED FEATURE-LEVEL FUSION OF VERY HIGH RESOLUTION AERIAL IMAGERY AND LIDAR DATA

Abstract. Land-cover classification of Remote Sensing (RS) data in urban area has always been a challenging task due to the complicated relations between different objects. Recently, fusion of aerial imagery and light detection and ranging (LiDAR) data has obtained a great attention in RS communities. Meanwhile, convolutional neural network (CNN) has proven its power in extracting high-level (deep) descriptors to improve RS data classification. In this paper, a CNN-based feature-level framework is proposed to integrate LiDAR data and aerial imagery for object classification in urban area. In our method, after generating low-level descriptors and fusing them in a feature-level fusion by layer-stacking, the proposed framework employs a novel CNN to extract the spectral-spatial features for classification process, which is performed using a fully connected multilayer perceptron network (MLP). The experimental results revealed that the proposed deep fusion model provides about 10% improvement in overall accuracy (OA) in comparison with other conventional feature-level fusion techniques.