The impact of climate on construction site safety varies significantly depending on subcontractor types due to the diverse nature of workplaces and work methods. This study introduces a novel approach by categorizing construction work according to subcontractor types and assessing accident risk probabilistically through the Physiologically Equivalent Temperature (PET), an outdoor thermal comfort index. Additionally, a Hidden Markov Model (HMM)-based clustering methodology was proposed to classify new groups using PET and accident probability. This study proceeded in the following sequence: (i) collection and classification of data, (ii) PET calculation, (iii) calculation of accident probability, and (iv) clustering and Pearson correlation coefficient analysis. As a result of clustering, each group was classified according to the workplace. Groups 2 and 3 demonstrated a strong positive correlation between accident probability and PET, with correlation coefficients of 0.837 and 0.772, while Group 1 exhibited a moderately positive correlation of 0.474. This study quantitatively evaluated the impact of climate on workers for each subcontractor type using PET, an outdoor thermal comfort index for construction work, and accident probability, resulting in the identification of new groups. The findings of this study may serve as novel benchmarks for safety management in construction worker safety based on PET.