Construction workers frequently experience mental fatigue owing to the high cognitive load of their tasks in a dynamic, complex environment, diminishing their cognitive ability and mobility and necessitating monitoring to ensure safety. Traditional fatigue evaluations rely on the subjective judgment of site managers and workers; this study developed a wireless device to objectively monitor the mental fatigue of construction workers using electroencephalogram (EEG) signals. The EEG signals of 16 construction workers were recorded while they performed a continuous physical–cognitive task, and the resulting EEG time–frequency–energy data were processed by a continuous wavelet transform and convolutional neural network to identify mental fatigue states without requiring manual feature extraction. The cognitive fatigue state classifications provided by the proposed framework were shown to match the self-reported fatigue states with 88.85% accuracy. This method can therefore facilitate real-time, continuously updating fatigue identification to support safety management on construction sites.