Score Fusion Research Articles

Multi-resolution dictionary learning method based on sample expansion and its application in face recognition

In recent years, dictionary learning method has been widely applied to face recognition and achieved good performance. However, most dictionary learning methods have two problems. First, they focused on only the resolution of the original images and did not consider impact of different resolutions on dictionary performance. When the attained dictionary is used to solve practical application problems, the recognition result of real-world images that may have a large difference in resolution with the original images will be very disappointed. Second, function of the dictionary will decrease due to insufficient training samples. Considering the above problems, this paper proposes a multi-resolution dictionary learning method based on sample expansion. We convert the original images to different resolutions and generate a dictionary for each resolution. Similarly, a dictionary is also produced for each resolution of reasonable virtual images generated by the original images. Then, for a test sample, a simple and efficient score fusion scheme is used to combine scores of the original image and the virtual image to obtain the ultimate classification score. We have performed experiments on multiple face databases, and the results show that our method has better performance than some state-of-the-art methods.

A weighted exponential discriminant analysis through side-information for face and kinship verification using statistical binarized image features

Side-information based exponential discriminant analysis (SIEDA) is more efficient than side-information based linear discriminant analysis (SILDA) in computing the discriminant vectors because it maximizes the Fisher criterion function. In this paper, we develop a novel criterion, named side-information based weighted exponential discriminant analysis (SIWEDA), that is based on the classical SIEDA method. We reformulate and generalize the classical Fisher criterion function in order to maximize it, with the property to pull as close as possible the intra-class samples (within-class samples), and push and repulse away as far as possible the inter-class samples (between-class samples). Thus, SIWEDA selects the eigenvalues of high significance and eliminate those with less discriminative information. To reduce the feature vector dimensionality and lighten the class intra-variability, we use SIWEDA and within class covariance normalization (WCCN) using the proposed statistical binarized image features (StatBIF). Moreover, we use score fusion strategy to extract the complementarity of different weighting scales of our StatBIF descriptor. We conducted experiments to evaluate the performance of the proposed method under unconstrained environment, using five datasets namely LFW, YTF, Cornell KinFace, UB KinFace and TSKinFace datasets, in the context of matching faces and kinship verification in the wild conditions. The experiments showed that the proposed approach outperforms the current state of the art. Very interestingly, our approach showed superior performance compared to methods based on deep metric learning.

Investigating of nodes and personal authentications utilizing multimodal biometrics for medical application of WBANs security

The authentication of the Wireless Body Area Networks (WBANs) nodes is a vital factor in its medical applications. This paper, investigates methods of authentication over these networks. Also, an effective unimodal and multimodal biometrics identification approaches based on individual face and voice recognition or combined using different fusion types are presented. The cryptography and non-cryptography-based authentication are discussed in this research work and its suitability with the medical applications. Cryptographic based authentication is not suitable for WBANs. The biometrics authentication is discussed and its challenges. In this work, different fusion types in multimodal biometric are presented. There are two unimodal schemes have been presented based on using the voice and face image individually, these two biometrics have been used in the multimodal biometric scheme. The presneted multimodal scheme is evaluated and applied using the feature and score fusion. The mechanism operation of presented algorithm starts with capturing the biometics signals ‘Face/Voice’, the second step is the feature extracting from each biometric individually. The Artificial Neural Network (ANN), The Support Vector Machine (SVM) and the Gaussian Mixture Model (GMM) classifiers have been employed to perform the classification process individually. The computer simulation experiments reveal that the cepstral coefficients and statistical coefficients for voice recognition performed better for the voice scenario. Also, the Eigenface and support vector machine tools in the face recognition scheme performed better than other schemes. The multimodal results better than the unimodal schemes. Also, the results of the scores fusion-based multimodal biometric scheme is better than the feature fusion-based scheme. Hence, the biometric-based authentication is effective and applicable for the WBANs authentication and personality continuous authentication on these medical applications wireless networks.

Effectiveness of symmetric rejection for a secure and user convenient multistage biometric system

A multistage biometric verification system uses multiple biometrics and/or multiple biometric verifiers to generate a verification decision. The core of a multistage biometric verification system is reject option which allows a stage not to give a genuine/impostor decision when it is not confident enough. This paper studies the effectiveness of symmetric rejection for multistage biometric verification systems. The symmetric rejection method determines the reject region by symmetrically rejecting equal proportion of genuine and impostor scores. The applicability of a multistage biometric verification system depends on how secure and user convenient it is, which is measured by the performance–cost trade-off. This paper analyzes the performance–cost trade-off of symmetric rejection method by conducting extensive experiments. Experiments are performed on two biometric databases: (1) publicly available NIST database and (2) a keystroke database. In addition, the symmetric rejection method is empirically compared with two existing rejection methods: (1) sequential probability ratio test-based method, which uses score-fusion and (2) Marcialis et al.’s method, which does not use score fusion. Results demonstrate strong effect of symmetric rejection method on creating a secure and user convenient multistage biometric verification system.

Multimodal score level fusion for recognition using face and palmprint

The paper proposes a t-norm-based matching score fusion approach for a multimodal heterogenous biometric recognition system. Two trait-based multimodal recognition system is developed by using biometrics traits like palmprint and face. First, palmprint and face are pre-processed, extracted features and calculated matching score of each trait using correlation coefficient and combine matching scores using t-norm based score level fusion. Face database like Face 94, Face 95, Face 96, FERET, FRGC and palmprint database like IITD are operated for training and testing of algorithm. The results of experimentation show that the proposed algorithm provides the Genuine Acceptance Rate (GAR) of 99.7% at False Acceptance Rate (FAR) of 0.1% and GAR of 99.2% at FAR of 0.01% significantly improves the accuracy of a biometric recognition system. The proposed algorithm provides the 0.53% more accuracy at FAR of 0.1% and 2.77% more accuracy at FAR of 0.01%, when compared to existing works.

Attention and Feature Selection for Automatic Speech Emotion Recognition Using Utterance and Syllable-Level Prosodic Features

This work attempts to recognize emotions from human speech using prosodic information represented by variations in duration, energy, and fundamental frequency ( $$F_{0}$$ ) values. For this, the speech signal is first automatically segmented into syllables. Prosodic features at the utterance (15 features) and syllable level (10 features) are extracted using the syllable boundaries and trained separately using deep neural network classifiers. The effectiveness of the proposed approach is demonstrated on German speech corpus-EMOTional Sensitivity ASistance System (EmotAsS) for people with disabilities, the dataset used for the Interspeech 2018 Atypical Affect Sub-Challenge. The initial set of prosodic features on evaluation yields an unweighted average recall (UAR) of 30.15%. A fusion of the decision scores of these features with spectral features gives a UAR of 36.71%. This paper also employs methods like attention mechanism and feature selection using resampling-based recursive feature elimination (RFE) to enhance system performance. Implementing attention and feature selection followed by a score-level fusion improves the UAR to 36.83% and 40.96% for prosodic features and overall fusion, respectively. The fusion of the scores of the best individual system of the Atypical Affect Sub-Challenge and the proposed system provides a UAR (43.71%) above the best test result reported. The effectiveness of the proposed system has also been demonstrated on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database with a UAR of 63.83%.

Unsupervised Speech Signal-to-Symbol Transformation for Language Identification

This paper presents a new approach for unsupervised segmentation and labeling of acoustically homogeneous segments from the speech signals. The virtual labels, thus obtained, are used to build unsupervised acoustic models in the absence of manual transcriptions. We refer to this approach as unsupervised speech signal-to-symbol transformation. This approach mainly involves three steps: (i) segmenting the speech signal into acoustically homogeneous regions, (ii) assigning consistent labels to the acoustic segments with similar characteristics and (iii) iterative modeling of the acoustic segments sharing the same label. This work focuses on improving initial segmentation and acoustic segment labeling. A new kernel-Gram matrix-based approach is proposed for segmentation. The number of segments is automatically determined using this approach, and performance comparable to the state-of-the-art algorithms is achieved. The segment labeling is formulated in a graph clustering framework. Graph clustering methods require extensive computational resources for large datasets. A new graph growing-based strategy is proposed to make the algorithm scalable. A two-stage iterative modeling is used to refine the segment boundaries and segment labels alternately. The proposed method achieves highest normalized mutual information and purity on TIMIT dataset. Quality assessment of the virtual labels is performed by building a language identification (LID) system for Indian languages. A bigram language model is built using these virtual phones. The LID system built using these virtual labels and corresponding language model performs very close to the system trained using manual labels and an i-vector-based LID system. The fusion of unsupervised LID system scores from our approach and the i-vector approach outperforms the LID system built under the supervision of manual labels by a relative margin of 31.19% demonstrating the effectiveness of unsupervised LID systems that can be at par with supervised systems by using virtual labels.

Multi-view region-adaptive multi-temporal DMM and RGB action recognition

Human action recognition remains an important yet challenging task. This work proposes a novel action recognition system. It uses a novel multi-view region-adaptive multi-resolution-in-time depth motion map (MV-RAMDMM) formulation combined with appearance information. Multi-stream 3D convolutional neural networks (CNNs) are trained on the different views and time resolutions of the region-adaptive depth motion maps. Multiple views are synthesised to enhance the view invariance. The region-adaptive weights, based on localised motion, accentuate and differentiate parts of actions possessing faster motion. Dedicated 3D CNN streams for multi-time resolution appearance information are also included. These help to identify and differentiate between small object interactions. A pre-trained 3D-CNN is used here with fine-tuning for each stream along with multi-class support vector machines. Average score fusion is used on the output. The developed approach is capable of recognising both human action and human–object interaction. Three public-domain data-sets, namely MSR 3D Action, Northwestern UCLA multi-view actions and MSR 3D daily activity, are used to evaluate the proposed solution. The experimental results demonstrate the robustness of this approach compared with state-of-the-art algorithms.

Elderly fall detection based on multi-stream deep convolutional networks

Fall is the biggest threat to seniors, with significant emotional, physical and financial implications. It is the major cause of serious injuries, disabilities, hospitalizations and even death especially for elderly people living alone. Timely detection could provide immediate medical service to the injured and avoid its harmful consequences. Great number of vision-based techniques has been proposed by installing cameras in several everyday environments. Recently, deep learning has revolutionized these techniques, mostly using convolutional neural networks (CNNs). In this paper, we propose weighted multi-stream deep convolutional neural networks that exploit the rich multimodal data provided by RGB-D cameras. Our method detects automatically fall events and sends a help request to the caregivers. Our contribution is three-fold. We build a new architecture composed of four separate CNN streams, one for each modality. The first modality is based on a single combined RGB and depth image to encode static appearance information. RGB image is used to capture color and texture and depth image deals with illumination variations. In contrast of the first feature that lacks the contextual information about previous and next frames, the second modality characterizes the human shape variations. After background subtraction and person recognition, human silhouette is extracted and stacked to define history of binary motion HBMI. The last two modalities are used to more discriminate the motion information. Stacked amplitude and oriented flow are used in addition to stacked optical flow field to describe respectively the velocity, the direction and the motion displacements. The main motivation behind the use of these multimodal data is to combine complementary information such as motion, shape, RGB and depth appearance to achieve more accurate detection than using only one modality. Our second contribution is the combination of the four streams to generate the final decision for fall detection. We evaluate early and late fusion strategies and we have defined the weight of each modality based on its overall system performance. Weighted score fusion is finally adopted based on our experiments. In the third contribution, transfer learning and data augmentation are applied to increase the amount of training data, avoid over fitting and improve the accuracy. Experiments have been conducted on publicly available standard datasets and demonstrate the effectiveness of the proposed method compared to existing methods.

Combining Multiple Biometric Traits Using Asymmetric Aggregation Operators for Improved Person Recognition

Biometrics is a scientific technology to recognize a person using their physical, behavior or chemical attributes. Biometrics is nowadays widely being used in several daily applications ranging from smart device user authentication to border crossing. A system that uses a single source of biometric information (e.g., single fingerprint) to recognize people is known as unimodal or unibiometrics system. Whereas, the system that consolidates data from multiple biometric sources of information (e.g., face and fingerprint) is called multimodal or multibiometrics system. Multibiometrics systems can alleviate the error rates and some inherent weaknesses of unibiometrics systems. Therefore, we present, in this study, a novel score level fusion-based scheme for multibiometric user recognition system. The proposed framework is hinged on Asymmetric Aggregation Operators (Asym-AOs). In particular, Asym-AOs are estimated via the generator functions of triangular norms (t-norms). The extensive set of experiments using seven publicly available benchmark databases, namely, National Institute of Standards and Technology (NIST)-Face, NIST-Multimodal, IIT Delhi Palmprint V1, IIT Delhi Ear, Hong Kong PolyU Contactless Hand Dorsal Images, Mobile Biometry (MOBIO) face, and Visible light mobile Ocular Biometric (VISOB) iPhone Day Light Ocular Mobile databases have been reported to show efficacy of the proposed scheme. The experimental results demonstrate that Asym-AOs based score fusion schemes not only are able to increase authentication rates compared to existing score level fusion methods (e.g., min, max, t-norms, symmetric-sum) but also is computationally fast.

Singular value decomposition-based virtual representation for face recognition

In the field of face recognition, a key issue is whether there are a sufficient number of face training samples with valid information. Due to the complexity of human face images, face recognition is easy to be affected by the external environment such as light intensity, gesture expression, hairstyle, and occlusion. Therefore, it is difficult to obtain enough effective samples in practical applications. In this paper, we propose a new algorithm that generates virtual images by utilizing the information of the test sample via singular value decomposition. The virtual images not only extend the training sample set but also can better adapt to the test sample. In addition, we use the weighted score fusion scheme to calculate the ultimate result, which can better take advantages of data from different sources including original images and virtual images. Experimental results on the Extended Yale_B, AR, GT, ORL, and FERET face databases prove that our algorithm can obtain satisfactory performance.

Weighted quasi‐arithmetic mean based score level fusion for multi‐biometric systems

Biometrics is now being principally employed in many daily applications ranging from the border crossing to mobile user authentication. In the high-security scenarios, biometrics require stringent accuracy and performance criteria. Towards this aim, multi-biometric systems that fuse the evidences from multiple sources of biometric have exhibited to diminish the error rates and alleviate inherent frailties of the individual biometric systems. In this article, a novel scheme for score-level fusion based on weighted quasi-arithmetic mean (WQAM) has been proposed. Specifically, WQAMs are estimated via different trigonometric functions. The proposed fusion scheme encompasses properties of both weighted mean and quasi-arithmetic mean. Moreover, it does not require any leaning process. Experimental results on three publicly available data sets (i.e. NIST-BSSR1 Multimodal, NIST-BSSR1 Fingerprint and NIST-BSSR1 Face) for multi-modal, multi-unit and multi-algorithm systems show that presented WQAM fusion algorithm outperforms the previously proposed score fusion rules based on transformation (e.g. t-norms), classification (e.g. support vector machines) and density estimation (e.g. likelihood ratio) methods.

Improved image representation and sparse representation for image classification

It seems that for multiple available images of the same object, the pixel values at the same image position are almost always different, which is especially obvious for the deformable object. This implies that it will be not easy to correctly classify the deformable object. In order to extract salient features of images and improve the performance of image classification, a novel image classification algorithm is proposed in this paper. The algorithm can effectively preserve the large-scale information and global features of the original image, reduce the difference in different images of the same object, and significantly improve the accuracy of image classification. Firstly, the virtual image is generated by the new image representation procedure. Secondly, the image classification algorithm is used to obtain the corresponding classification scores of the original image and the virtual image, respectively. Finally, the ultimate classification score is obtained by a simple and efficient score fusion scheme. A large number of experiments on three widely used image databases show that the proposed algorithm outperforms other state-of-the-art algorithms in classification accuracy. At the same time, the algorithm has the advantages of simple implementation and high computational efficiency.

A study on deep learning spatiotemporal models and feature extraction techniques for video understanding

Video understanding requires abundant semantic information. Substantial progress has been made on deep learning models in the image, text, and audio domains, and notable efforts have been recently dedicated to the design of deep networks in the video domain. We discuss the state-of-the-art convolutional neural network (CNN) and its pipelines for the exploration of video features, various fusion strategies, and their performances; we also discuss the limitations of CNN for long-term motion cues and the use of sequential learning models such as long short-term memory to overcome these limitations. In addition, we address various multi-model approaches for extracting important cues and score fusion techniques from hybrid deep learning frameworks. Then, we highlight future plans in this domain, recent trends, and substantial challenges for video understanding. This survey’s objectives are to study the plethora of approaches that have been developed for solving video understanding problems, to comprehensively study spatiotemporal cues, to explore the various models that are available for solving these problems and to identify the most promising approaches.

A four-stream ConvNet based on spatial and depth flow for human action classification using RGB-D data

Appearance and depth-based action recognition has been researched exclusively for improving recognition accuracy by considering motion and shape recovery particulars from RGB-D video data. Convolutional neural networks (CNN) have shown evidences of superiority on action classification problems with spatial and apparent motion inputs. The current generation of CNNs use spatial RGB videos and depth maps to recognize action classes from RGB-D video. In this work, we propose a 4-stream CNN architecture that has two spatial RGB-D video data streams and two apparent motion streams, with inputs extracted from the optical flow of RGB-D videos. Each CNN stream is packed with 8 convolutional layers, 2 dense and one SoftMax layer, and a score fusion model to merge the scores from four streams. Performance of the proposed 4-stream action recognition framework is tested on our own action dataset and three benchmark datasets for action recognition. The usefulness of the proposed model is evaluated with state-of-the-art CNN architectures for action recognition.

High-Resolution Remote Sensing Image Retrieval Based on Classification-Similarity Networks and Double Fusion

In high-resolution remote sensing image retrieval (HRRSIR), convolutional neural networks (CNNs) have an absolute performance advantage over the traditional hand-crafted features. However, some CNN-based HRRSIR models are classification-oriented, they pay no attention to similarity, which is critical to image retrieval; whereas others concentrate on learning similarity, failing to take full advantage of information about class labels. To address these issues, we propose a novel model called classification-similarity network (CSN) , which aims for image classification and similarity prediction at the same time. In order to further improve performance, we build and train two CSNs, and two kinds of information from them, i.e., deep features and similarity scores, are consolidated to measure the final similarity between two images. Besides, the optimal fusion theorem in biometric authentication, which gives a theoretical scheme to make sure that fusion will definitely lead to a better performance, is used to conduct score fusion. Extensive experiments are carried out over publicly available datasets, demonstrating that CSNs are distinctly superior to usual CNNs and our proposed “two CSNs + feature fusion + score fusion” method outperforms the state-of-the-art models.

Predicting the Helpfulness Score of Product Reviews Using an Evidential Score Fusion Method

Everyday many online product sales websites and specialized reviewing forums publish a massive volume of human-generated product reviews. People use these reviews as valuable free source of knowledge when decide to buy products. Therefore, an accurate automated system for distinguishing useful reviews from non-useful ones is of great importance. This article presents a new model for specifying the usefulness of comments using the textual features extracted from the reviews. Various types of features including emotion-related, linguistic and text-related features, valence, arousal, and dominance (VAD) values, review-length and polarity of comments are exploited in this study. Moreover, two new algorithms are presented: an improved evidential algorithm for emotion recognition, and an algorithm for extracting VAD values for each review. Finally, the usefulness of reviews is predicted using the mentioned features and an improved Dempster-Shafer score fusion algorithm. The proposed method is applied to review datasets of Books and Video Games of Amazon. The results show that combining the features associated with emotions, features of VAD, and text-related features improves the accuracy of predicting the usefulness of reviews. Also, in comparison with the original Dempster-Shafer method, the precision of the improved Dempster-Shafer algorithm for both datasets is 15% and 11% higher, respectively.

A Watermarking Based Approach for Protection of Templates in Multimodal Biometric System

Template protection is an essential requirement in biometric based authentication systems due to security and privacy issues. These issues have been arisen because of unsafe storage and misuse of templates. Techniques for template protection are developed so that templates stored in databases remain safe and inaccessible to unauthorized users. Security and privacy are the major factors to be achieved through these techniques. This paper presents a watermarking based approach to protect templates in multimodal biometric systems. Here, iris and face biometric traits have been used. This approach has used watermarking technique to embed iris image into face image to allow their safe storage in database. At the time of authentication, both iris and face patterns are retrieved from the watermarked image and matching is performed with query images. The final decision is given by matching score obtained after fusion of iris and face matching scores at match score level. MATLAB has been used to implement the proposed approach and the observation from results confirms that it does not degrade the recognition performance of existing multimodal biometric system. Rather, it provides protection to templates and facilitates to reject the compromised templates by extending the decomposition to higher level or simply by changing the embedding and extraction algorithms.

Multi-modal Authentication Using Score Fusion of ECG and Fingerprints

Multi-modal Authentication Using Score Fusion of ECG and Fingerprints

Low complexity high efficiency coding of light fields using ensemble classifiers

Light field images can be efficiently compressed using standard video codecs, such as the High Efficiency Video Coding (HEVC). However, the huge amount of data combined with the high computational complexity of HEVC, poses limitations on high-speed light field capturing and storage. This paper presents a contribution for low complexity encoding of light fields, in different formats using HEVC, based on a Random Forests ensemble classifier. Optimal features for training the classifier are found through a score fusion based approach. Using the HEVC still image profile, the proposed method gives speed-up of 56.23% for sub-aperture images. For pseudo video format, the proposed method outperforms others available in the literature, yielding an average speed-up of 62.18%, 56.54% and 44.73% for Random Access, Low-delay Main and All-Intra profiles respectively, with negligible decrease in RD performance. These are novel results in fast coding of light fields, which are useful for further research and benchmarking.