This work aims to develop a subject-independent Emotion Detection System (EDS) based on EEG signals and the 3D Valence-Arousal-Dominance (VAD) model. The DEAP database physiological signals are considered for the system design. A multi-domain feature extraction is performed using the Wavelet-based Atomic Function time–frequency domain technique; and various time and frequency domain feature extraction techniques. Further, principal component analysis reduces the data dimensionality and redundancy in the obtained feature set. The minimal feature set is analysed using machine learning classifiers, i.e., gradient boosting, decision tree, and random forest. Additionally, the hyperparameters of machine learning algorithms are tuned using Optuna to improve the performance of the proposed model. Three EDS are designed in this work; EDS1 considers the 3D VAD model for three class classifications. 2D VA model is used in EDS2 to determine 9 discrete emotions, and EDS3 detects 12 discrete emotions using the 3D VAD model. Results reveal the highest classification accuracy of about 99% with EDS1, whereas an average accuracy of 99.82% and 98.44% is obtained with EDS2 and EDS3, respectively. The results reveal that both EDS1 and EDS2 show improvement as compared to the existing literature. Also, the proposed EDS3 provides the classification of the highest number of discrete emotions, which may facilitate the design of an efficient human–computer interface system.