Emotion based Contextual Semantic Relevance Feedback in Multimedia Information Retrieval

Abstract

Every query issued by a user to find some relevant information,contains the semantic and its associated contexts, but, indentifying and conveying these semantic and context (present in the query) to MIR system is a major challenge and still needs to be tackled effectively. Thus, exploiting the plausibility of context associated with semantic concept for the purpose of enhancement in retrieval of the possible relevant information, we propose Emotion Based Contextual Semantic Relevance Feedback(ECSRF) to learn, refine, discriminate and identify the current context present in a query. We will further investigate: (1) whether multimedia attributes(audio, speech along with visual) can be purposefully used to work out a current context of user’s query and will be useful inreduction of search space and retrieval time; (2) whether increasing the Affective features (spoken emotional word(s)with facial expression(s)) in identifying, discriminating emotions would increase the overall retrieval performance in terms of Precision, Recall and retrieval time;(3)whether increasing the discriminating power of classifier algorithm in query perfection would increase the search accuracy with less retrieval time.We introduce an Emotion Recognition Unit(ERU) that comprises of a customized 3D spatiotemporal Gabor filter to capture spontaneous facial expression, and emotional word recognition system (combination of phonemes and visemes) to recognize the spoken emotional words. Integration of classifier algorithms GMM, SVM and CQPSB are compared in ECSRF framework to study the effect of increasing the discriminating power of classifier on retrieval performance.Observations suggest that prediction of contextual semantic relevance is feasible, and ECSRF model can benefit from incorporating such increased affective features and classifier to increase a MIR system’s retrieval efficiency and contextual perceptions.