Keyframe Selection Framework Based on Visual and Excitement Features for Lifelog Image Sequences

Abstract

Keyframe selection is the process of finding a set of representative frames from an image sequence. We aim to achieve an automatic keyframe selection. The main problem is that the accuracy of keyframe selection is highly subjective to each particular user. To deal with this problem, we propose a VIEMO keyframe selection framework based on visual and excitement features that consists of two integrated modules, namely; event segmentation and keyframe selection. Firstly, scene change detection algorithm was applied for an event segmentation. Later, visual features which are contrast, color variance, sharpness, noise and saliency along with excitement features from a biosensor are used to filter keyframe that closely matches with user selection keyframe. Two different fusion scheme which are flat and hierarchical fusion were also investigated. To evaluate the quality of keyframe from the proposed method, we present an evaluation techniques which grades the quality of the keyframe automatically. Even when the keyframe does not exactly match with the keyframe selected by the user, the degree of acceptance calculated from visual similarity is provided. Experimental results showed that keyframe selection using only visual features yielded an acceptance rate of \(74.16 \,\%\). Our proposed method achieves a higher acceptance rate of \(83.71\,\%\). Moreover, the acceptance rate was improved by the average of \(9.55\,\%\) in all participants. Therefore, our framework provides a potential solution to this subjective issue for keyframe selection in lifelog image sequences selection.