Robust Clustering-Based Automated Video Shot Boundary Detection Using Handcrafted and Deep Feature Fusion

Abstract

Video shot boundary detection (VSBD) plays a key role in analyzing, summarizing, indexing and retrieving content-based data from videos. Many artificial intelligence (AI)-based techniques have recently been introduced to detect the gradual transition from video frames. However, those techniques fail to detect the presence of inter-classes like fade-in, fade-out and dissolve from gradual transition video frames. In addition, the existing techniques face high computational complexity during detecting transitions from the video frames. This research brings novel clustering-based techniques to classify the inter-classes like fade-in, fade-out and dissolve from gradual transition video frames. At the initial stage, color histogram differences (CHD) technique is introduced to detect the abrupt transition from the video frames. The identified abrupt transitions are completely removed from the video frames. Then, segmentation is done to segment the gradual transitions from the video frames. The segmented gradual transition frames are then given to extract the handcrafted and deep features from the video frames. The deep features are extracted from the segments using the Morlet Wavelet-assisted modified stacked autoencoder (MW-MSAE) technique. The extracted handcrafted features and deep features are then concatenated together, and finally, fused feature vectors are obtained. The fused features are then fed to the robust deep [Formula: see text]-means map clustering method (RDKMM) to aggregate based on similar features. For calculating the similar features, a similarity-based correlation calculation (SBCC) is done in adjacent frames to determine gradual shot transitions like fade-in, fade-out and dissolve. The dataset used in this research is the TREC Video Retrieval Evaluation (TRECVID) 2021 dataset. In the experimental scenario, an accuracy of 95.5%, a sensitivity of 95.3%, a specificity of 96.9%, a precision of 93.9%, an F-measure of 94.6% and Mathew’s correlation coefficient (MCC) of 92.4% are obtained.