Multi-modal authentication system based on audio-visual data

Abstract

Biometrics is an emerging technology in this era, which has been widely used in many application such as secured access to a computer and any other system, criminal identification, person authentication, etc. Face recognition is a biometric method of identifying a person by comparing the data with the stored information of that person. But the recognition of face can be affected by illumination variation, facial expression and other issues. Therefore, the authentication using only face image might be difficult for the system. To overcome this challenge, biometric authentications have to rely on more than one method. In this paper, we consider audio-video person authentication based on two sources of information. Single modality evidence has limitations in both security and robustness. Therefore, here, audio recognition is added with visual recognition. Facial features, as well as speech is extracted separately from audio-visual data and integrate both the modality for secure user authentication. Mel Frequency Cepstral Coefficients (MFCC) is used as the speech feature. Viola-Jones and Scale-Invariant Feature Transform (SIFT) algorithm are used for visual feature extraction. After the extraction of audio and visual features, the feature selection method is employed. The two-phase algorithm consisting of statistical analysis, Analysis of Variance (ANOVA) with Incremental Feature Selection (IFS) is proposed to select significant features from audio-visual data. The Audio and Video processing is done in two separate phases using machine learning algorithms. The results of both the modalities are then combined at the decision level based on majority voting. It has been observed that multiple modalities of both audio-visual information give immensely good results compared to a standalone single modality.