Classification of electroencephalography (EEG) signals associated with Steady-state visually evoked potential (SSVEP) is prominent because of its potential in restoring the communication and controlling capability of paralytic people. However, SSVEP signals classification is a challenging task for researchers because of its low signal-to-noise ratio, non-stationary and high dimensional properties. A proficient technique has to be evolved to classify the SSVEP-based EEG data. In recent times, convolutional neural network (CNN) has reached a quantum leap in EEG signal classification. Therefore, the proposed system employs CNN to classify the SSVEP-based EEG signals. Though CNN has proved its proficiency in handling EEG signal classification problems, the calibration of hyperparameters is required to enhance the performance of the model. The calibration of a hyperparameter is a time-consuming task, hence proposed an automated hyperparameter optimization technique using the Red Fox Optimization Algorithm (RFO). The effectiveness of the algorithm is evaluated by comparing it with the performance of Harris Hawk Optimization (HHO), Flower Pollination Algorithm (FPA), Grey Wolf Optimization Algorithm (GWO) and Whale Optimization Algorithm (WOA) based hyperparameter optimized CNN applied to the SSVEP based EEG signals multiclass dataset. The experimental results infer that the proposed algorithm can achieve a testing accuracy of 88.91% which is higher than other comparative algorithms like HHO, FPA, GWO and WOA. The above-mentioned values clearly show that the proposed algorithm achieved competitive performance when compared to the other reported algorithm.