Multimodal Recognition Research Articles

A multi-modal and multi-scale emotion-enhanced inference model based on fuzzy recognition

Only the label corresponding to the maximum value of the fully connected layer is used as the output category when a neural network performs classification tasks. When the maximum value of the fully connected layer is close to the sub-maximum value, the classification obtained by considering only the maximum value and ignoring the sub-maximum value is not completely accurate. To reduce the noise and improve classification accuracy, combining the principles of fuzzy reasoning, this paper integrates all the output results of the fully connected layer with the emotional tendency of the text based on the dictionary to establish a multi-modal fuzzy recognition emotion enhancement model. The provided model considers the enhancement effect of negative words, degree adverbs, exclamation marks, and question marks based on the smallest subtree on the emotion of emotional words, and defines the global emotional membership function of emojis based on the corpus. Through comparing the results of CNN, LSTM, BiLSTM and GRU on Weibo and Douyin, it is shown that the provided model can effectively improve the text emotion recognition when the neural network output result is not clear, especially for long texts.

Human Motion Recognition Based on Multimodal Characteristics of Learning Quality in Football Scene

The task of human motion recognition based on video is widely concerned, and its research results have been widely used in intelligent human-computer interaction, virtual reality, intelligent monitoring, security, multimedia content analysis, etc. The purpose of this study is to explore the human action recognition in the football scene combined with learning quality related multimodal features. The method used in this study is to select BN-Inception as the underlying feature extraction network and use uncontrolled environment and real world to capture datasets UCFl01 and HMDB51, and pretraining is carried out on the ImageNet dataset. The spatial depth convolution network takes image frame as input, and the temporal depth convolution network takes stacked optical flow as input to carry out human action multimodal identification. In the results of multimodal feature fusion, the accuracy of UCFl01 dataset is generally high, all of which are over 80%, and the highest is 95.2%, while the accuracy of HMDB51 dataset is about 70%, and the lowest is only 56.3%. It can be concluded that the method of this study has higher accuracy and better effect in multimodal feature acquisition, and the accuracy of single-mode feature recognition is significantly lower than that of multimodal feature recognition. It provides an effective method for the multimodal feature of human motion recognition in the scene of football or sports.

Single Sample Discriminant Analysis Based on Gabor Transform

To solve the small sample problem of biometric identification, this paper investigates the limiting case of the problem, i.e., the recognition of a single training sample, and proposes a single sample discriminant analysis method based on Gabor wavelet and KPCA-RBF (KPRC) classifier (kernel principal component analysis-radial basis function). The proposed method performs pixel-level fusion of face and palmprint images. Firstly, a face image and a palmprint image were subject to two-dimensional (2D) Gabor wavelet transform. The resulting Gabor face image and Gabor palmprint image were fused on the pixel level into a new fused image. Next, a new classifier called KPCA-RBF was designed to extract nonlinear discriminative features by KPCA, and classify objects with RBF. Based on AR database, FERET database, and palmprint database, the single sample discriminant analysis method was realized based on Gabor transform and KPCA-RBF classifier. Experimental results show that multimodal recognition methods clearly outshine single-modal recognition methods, and the GABOR-KPRC with pixel-level fusion achieves better recognition effect than other fusion methods. It was also demonstrated that Gabor transform and KPRC classifier can effectively improve the fusion effect, whether for pixel-level fusion or decision-level fusion.

A nature‐inspired meta‐heuristic paradigm for person identification using multimodal biometrics

AbstractBecause of the expanding improvement of the internet, security toward individual data has gotten more significant in recent decades. Due to the increasing development of the internet, security toward personal information has become more important in recent decades. Various person recognition methods are introduced for ensuring the security of a person's information. However identifying the information of the user using unique physiological characteristics poses a major issue in the biometric recognition system. Hence, an effective multimodal recognition system is developed using the proposed Elephant Deer Hunting Optimization‐based hybrid fusion (EDHO‐based hybrid fusion) model for person recognition. The proposed EDHO is designed by the integration of Elephant Herding Optimization and Deer Hunting Optimization Algorithm. Here, three different modalities, such as finger vein, dorsal hand vein, and electromyography (EMG) data are employed for the person recognition process. To increase the level of security, the hybrid feature fusion process is employed in such a way that features are fused based on the weight coefficients. The optimization algorithm is considered for the computation of the weight factor. However, the optimal weight value shows the optimal solution of person recognition in such a way that the computation of optimal value is based on the fitness measure. The proposed method achieved higher performance of accuracy as 0.963, True Negative Rate (TNR) as 0.9658, and True Positive rate (TPR) as 0.9548, respectively.

Analysis of different affective state multimodal recognition approaches with missing data-oriented to virtual learning environments

In this work, the affective state of users in virtual learning environments is assessed/recognized in terms of continuous arousal and valence dimensions, making use of multimodal information (audio, text and video), whenever any of these modalities are available. In general, virtual learning environments where these three modalities are all the time, are not common; at some moments only the video modality is available, while in others only text or/and video and/or audio. Different approaches using feature-level fusion and decision-level fusion are proposed for multimodal recognition with missing data. Recognizing according to available modalities is studied following the ideas of dropout from neural networks and of variable input length from recurrent neural networks. This proposal is innovative because it represents emotions in the continuous space, which is not common in virtual education; and makes use of the available modalities in a virtual environment in a given moment, which is very common in virtual learning environments because the people are not speaking or writing all the time.

Multimodal Low Resolution Face and Frontal Gait Recognition from Surveillance Video

Biometric identification using surveillance video has attracted the attention of many researchers as it can be applicable not only for robust identification but also personalized activity monitoring. In this paper, we present a novel multimodal recognition system that extracts frontal gait and low-resolution face images from frontal walking surveillance video clips to perform efficient biometric recognition. The proposed study addresses two important issues in surveillance video that did not receive appropriate attention in the past. First, it consolidates the model-free and model-based gait feature extraction approaches to perform robust gait recognition only using the frontal view. Second, it uses a low-resolution face recognition approach which can be trained and tested using low-resolution face information. This eliminates the need for obtaining high-resolution face images to create the gallery, which is required in the majority of low-resolution face recognition techniques. Moreover, the classification accuracy on high-resolution face images is considerably higher. Previous studies on frontal gait recognition incorporate assumptions to approximate the average gait cycle. However, we quantify the gait cycle precisely for each subject using only the frontal gait information. The approaches available in the literature use the high resolution images obtained in a controlled environment to train the recognition system. However, in our proposed system we train the recognition algorithm using the low-resolution face images captured in the unconstrained environment. The proposed system has two components, one is responsible for performing frontal gait recognition and one is responsible for low-resolution face recognition. Later, score level fusion is performed to fuse the results of the frontal gait recognition and the low-resolution face recognition. Experiments conducted on the Face and Ocular Challenge Series (FOCS) dataset resulted in a 93.5% Rank-1 for frontal gait recognition and 82.92% Rank-1 for low-resolution face recognition, respectively. The score level multimodal fusion resulted in 95.9% Rank-1 recognition, which demonstrates the superiority and robustness of the proposed approach.

Multi model biometrics Recognition using different transforms

To give secure support to biometrics recognition this system deals with multimodal recognition (finger print + Iris recognition) to give efficient performance in the biometrics recognition. The kaggle & MRL eye database is used for the image for both finger print and Iris recognition, 225X225 image size taken into the system and feature extraction made namely edge and color then, combine both the feature together and stored into database. The same procedure applicable for training image as well as testing image. Haar transform, Fourier Transform and Laplace Transform for the feature extraction used and compare the result among the transform result justify the efficiency of the best transform. Now, The testing image load into the system feature extraction made and match with the database for the secure process in multi model biometrics.

Data-driven modeling and analysis based on complex network for multimode recognition of industrial processes

An industrial process usually has multiple operating conditions or periods due to various factors such as the fluctuation of raw material quality, differences of worker levels, and changes of product specifications. Industrial data from different operating conditions or periods have different statistical characteristics, namely multimode behavior. In fact, multimode recognition of industrial processes is the first problem to develop effective intelligent manufacturing strategies for engineers. The clustering-based methods can help them characterize multimode behavior in historical databases and group them together with little detailed knowledge of the process. However, these methods are facing the challenges of massive high-dimensional industrial big data and unknown mode number. This study proposes a construction method of complex network to describe the potential relationship in industrial big data. The topological structure of complex network can be acquired by taking the correlation matrix of multi-variables as nodes and the similarity between correlation matrices as edges. On this basis, community detection technology is employed to reveal the aggregation behavior of nodes in network. It takes the maximum modularity as the optimization objective, and partitions each node into one community through greedy search algorithm, where a community denotes a mode of industrial process. Finally, case studies have demonstrated the proposed method has good performance and application prospects.

Multimodal biometrics of fingerprint and signature recognition using multi-level feature fusion and deep learning techniques

Providing security in biometrics is the major challenging task in the current situation. A lot of research work is going on in this area. Security can be more tightened by using complex security systems, like by using more than one biometric trait for recognition. In this paper multimodal biometric models are developed to improve the recognition rate of a person. The combination of physiological and behavioral biometrics characteristics is used in this work. Fingerprint and signature biometrics characteristics are used to develop a multimodal recognition system. Histograms of oriented gradients (HOG) features are extracted from biometric traits and for these feature fusions are applied at two levels. Features of fingerprint and signatures are fused using concatenation, sum, max, min, and product rule at multilevel stages, these features are used to train deep learning neural network model. In the proposed work, multi-level feature fusion for multimodal biometrics with a deep learning classifier is used and results are analyzed by a varying number of hidden neurons and hidden layers. Experiments are carried out on SDUMLA-HMT, machine learning and data mining lab, Shandong University fingerprint datasets, and MCYT signature biometric recognition group datasets, and encouraging results were obtained.

Correspondence Learning for Deep Multi-Modal Recognition and Fraud Detection

Deep learning-based methods have achieved good performance in various recognition benchmarks mostly by utilizing single modalities. As different modalities contain complementary information to each other, multi-modal based methods are proposed to implicitly utilize them. In this paper, we propose a simple technique, called correspondence learning (CL), which explicitly learns the relationship among multiple modalities. The multiple modalities in the data samples are randomly mixed among different samples. If the modalities are from the same sample (not mixed), then they have positive correspondence, and vice versa. CL is an auxiliary task for the model to predict the correspondence among modalities. The model is expected to extract information from each modality to check correspondence and achieve better representations in multi-modal recognition tasks. In this work, we first validate the proposed method in various multi-modal benchmarks including CMU Multimodal Opinion-Level Sentiment Intensity (CMU-MOSI) and CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) sentiment analysis datasets. In addition, we propose a fraud detection method using the learned correspondence among modalities. To validate this additional usage, we collect a multi-modal dataset for fraud detection using real-world samples for reverse vending machines.

Multimodal recognition of emotions in music and language

We investigated through electrophysiological recordings how music-induced emotions are recognized and combined with the emotional content of written sentences. Twenty-four sad, joyful, and frightening musical tracks were presented to 16 participants reading 270 short sentences conveying a sad, joyful, or frightening emotional meaning. Audiovisual stimuli could be emotionally congruent or incongruent with each other; participants were asked to pay attention and respond to filler sentences containing cities’ names, while ignoring the rest. The amplitude values of event-related potentials (ERPs) were subjected to repeated measures ANOVAs. Distinct electrophysiological markers were identified for the processing of stimuli inducing fear (N450, either linguistic or musical), for language-induced sadness (P300) and for joyful music (positive P2 and LP potentials). The music/language emotional discordance elicited a large N400 mismatch response ( p = .032). Its stronger intracranial source was the right superior temporal gyrus (STG) devoted to multisensory integration of emotions. The results suggest that music can communicate emotional meaning as distinctively as language.

Joint Discriminative Sparse Coding for Robust Hand-Based Multimodal Recognition

Multimodal biometrics recognition has recently attracted much interest for its higher security and effectiveness compared with unimodal biometrics recognition. However, most of the conventional multimodal recognition approaches generally focus on extracting semantic information from different modalities independently, while ignoring the implicit correlations among inter-modality. In this paper, we propose a simple yet effective supervised multimodal feature learning method, called joint discriminative sparse coding (JDSC), which is applied for hand-based multimodal recognition including finger-vein and finger-knuckle-print fusion, palm-vein and palmprint fusion, as well as palm-vein and dorsal-hand-vein fusion. Considering that relevant samples from different modalities have semantic correlations, JDSC projects the raw data into a shared space in which the distance of the between-class is maximized and the distance of the within-class is minimized, at the same time, the correlation among the inter-modality of the within-class is maximized. Therefore, sparse binary codes quantified by the obtained projection matrix can have more discriminative power for multimodal recognition tasks. Thorough experiments on six commonly used multimodal datasets demonstrate the superiority of our proposed method over several state-of-the-art techniques.

Research on a Microexpression Recognition Technology Based on Multimodal Fusion

Microexpressions have extremely high due value in national security, public safety, medical, and other fields. However, microexpressions have characteristics that are obviously different from macroexpressions, such as short duration and weak changes, which greatly increase the difficulty of microexpression recognition work. In this paper, we propose a microexpression recognition method based on multimodal fusion through a comparative study of traditional microexpression recognition algorithms such as LBP algorithm and CNN and LSTM deep learning algorithms. The method couples the separate microexpression image information with the corresponding body temperature information to establish a multimodal fusion microexpression database. This paper firstly introduces how to build a multimodal fusion microexpression database in a laboratory environment, secondly compares the recognition accuracy of LBP, LSTM, and CNN + LSTM networks for microexpressions, and finally selects the superior CNN + LSTM network in the comparison results for model training and testing on the test set under separate microexpression database and multimodal fusion database. The experimental results show that a microexpression recognition method based on multimodal fusion designed in this paper is more accurate than unimodal recognition in multimodal recognition after feature fusion, and its recognition rate reaches 75.1%, which proves that the method is feasible and effective in improving microexpression recognition rate and has good practical value.

UGaitNet: Multimodal Gait Recognition With Missing Input Modalities

UGaitNet: Multimodal Gait Recognition With Missing Input Modalities

Iris-Fingerprint multimodal biometric system based on optimal feature level fusion model

<abstract><p>For reliable and accurate multimodal biometric based person verification, demands an effective discriminant feature representation and fusion of the extracted relevant information across multiple biometric modalities. In this paper, we propose feature level fusion by adopting the concept of canonical correlation analysis (CCA) to fuse Iris and Fingerprint feature sets of the same person. The uniqueness of this approach is that it extracts maximized correlated features from feature sets of both modalities as effective discriminant information within the features sets. CCA is, therefore, suitable to analyze the underlying relationship between two feature spaces and generates more powerful feature vectors by removing redundant information. We demonstrate that an efficient multimodal recognition can be achieved with a significant reduction in feature dimensions with less computational complexity and recognition time less than one second by exploiting CCA based joint feature fusion and optimization. To evaluate the performance of the proposed system, Left and Right Iris, and thumb Fingerprints from both hands of the SDUMLA-HMT multimodal dataset are considered in this experiment. We show that our proposed approach significantly outperforms in terms of equal error rate (EER) than unimodal system recognition performance. We also demonstrate that CCA based feature fusion excels than the match score level fusion. Further, an exploration of the correlation between Right Iris and Left Fingerprint images (EER of 0.1050%), and Left Iris and Right Fingerprint images (EER of 1.4286%) are also presented to consider the effect of feature dominance and laterality of the selected modalities for the robust multimodal biometric system.</p></abstract>

Multimodal action recognition using variational‐based Beta‐Liouville hidden Markov models

Multimodal action recognition using variational‐based Beta‐Liouville hidden Markov models

Mathematical Representation of Emotion Using Multimodal Recognition Model with Deep Multitask Learning

Mathematical Representation of Emotion Using Multimodal Recognition Model with Deep Multitask Learning

Complex ISAR target recognition using deep adaptive learning

Target recognition in real-world environment is a most important and challenging computer vision task. In this paper, a real-world complex ISAR target recognition method is proposed on a computationally limited platform using deep adaptive learning (DAL). Particularly, adaptive multimodal mechanism (AMM) is presented to efficiently handle the complex multimodal recognition problem, which substantially improves CNNs’ sampling and transformation capability and significantly increases output feature maps’ resolutions. Moreover, feature learning, attribute prediction, relation mining, image matching, image understanding, image summarization, and label expansion are considered in a unified framework. Extensive qualitative and quantitative experiments are performed, and the results show the proposed method outperforms the several state-of-the-art methods.

Incremental approach for multi-modal face expression recognition system using deep neural networks

Facial expression recognition (FER) is still one of the most challenging tasks. Convolutional neural network (CNN) and deep convolutional neural network (DCNN) has evolved as an efficient tool for FER models, but they differ significantly in terms of their network configuration and architecture. There exists a variety of bottlenecks in existing FER systems, such as they lack in generalising their algorithms. In this paper, we propose a model based on DCNN to overcome these challenges. Firstly, the proposed model focuses on the selection of an appropriate activation function depending on its accuracy and training loss over a database. Secondly, an incremental strategy is used in which deeper models are developed simultaneously from shallower networks to increase the accuracy with less training loss. Lastly, by an ensemble of CNN and DCNNs, the model achieves an accuracy of 74.15% for FER2013, 96.20% for CK+, and 98.25% for JAFFE databases, outperforming previous work.

An Overview of Multimodal Biometrics Using the Face and Ear

In the recent years, we have witnessed the rapid development of face recognition, though it is still plagued by variations such as facial expressions, pose, and occlusion. In contrast to the face, the ear has a stable 3D structure and is nearly unaffected by aging and expression changes. Both the face and ear can be captured from a distance and in a nonintrusive manner, which makes them applicable to a wider range of application domains. Together with their physiological structure and location, the ear can readily serve as supplement to the face for biometric recognition. It has been a trend to combine the face and ear to develop nonintrusive multimodal recognition for improved accuracy, robustness, and security. However, when either the face or the ear suffers from data degeneration, if the fusion rule is fixed or with inferior flexibility, a multimodal system may perform worse than the unimodal system using only the modality with better quality sample. The biometric quality-based adaptive fusion is an avenue to address this issue. In this paper, we present an overview of the literature about multimodal biometrics using the face and ear. All the approaches are classified into categories according to their fusion levels. In the end, we pay particular attention to an adaptive multimodal identification system, which adopts a general biometric quality assessment (BQA) method and dynamically integrates the face and ear via sparse representation. Apart from a refinement of the BQA and fusion weights selection, we extend the experiments for a more thorough evaluation by using more datasets and more types of image degeneration.