Classification of Low- and High-Entropy File Fragments Using Randomness Measures and Discrete Fourier Transform Coefficients

Abstract

This paper presents an approach to improve the file fragment classification by proposing new features for classification and evaluating them on a dataset that includes both low- and high-entropy file fragments. High-entropy fragments, belonging to compressed and encrypted files, are particularly challenging to classify because they lack exploitable patterns. To address this challenge, the proposed feature vectors are constructed based on the byte frequency distribution (BFD) of file fragments, along with discrete Fourier transform coefficients and several randomness measures. These feature vectors are tested using three machine learning models: Support vector machines (SVMs), artificial neural networks (ANNs), and random forests (RFs). The proposed approach is evaluated on the govdocs1 dataset, which is freely available and widely used in this field, to enable reproducibility and fair comparison with other published research. The results show that the proposed approach outperforms existing methods and achieves better classification accuracy for both low- and high-entropy file fragments.