The atmospheric boundary layer (ABL) is a highly turbulent geophysical flow, which has chaotic and often too complex dynamics to unravel from limited data. Characterizing coherent turbulence structures in complex ABL flows under various atmospheric regimes is not systematically well established yet. This study aims to bridge this gap using large eddy simulations (LESs), Koopman theory, and unsupervised classification techniques. To this end, eight LESs of different convective, neutral, and unsteady ABLs are conducted. As the ratio of buoyancy to shear production increases, the turbulence structures change from roll vortices to convective cells. The quadrant analysis indicated that as this ratio increases, the sweep and ejection events decrease, and inward/outward interactions increase. The Koopman mode decomposition (KMD) is then used to characterize their turbulence structures. Our results showed that KMD can reveal non-trivial modes of highly turbulent ABL flows (e.g., transverse to the mean flow direction) and can reconstruct the primary dynamics of ABLs even under unsteady conditions with only ∼5% of the modes. We attributed the detected modes to the imposed pressure gradient (shear), Coriolis (inertial oscillations), and buoyancy (convection) forces by conducting novel timescale and quadrant analyses. We then applied the convolutional neural network combined with the K-means clustering to group the Koopman modes. This approach is displacement and rotation invariant, which allows efficiently reducing the number of modes that describe the overall ABL dynamics. Our results provide new insights into the dynamics of ABLs and present a systematic data-driven method to characterize their complex spatiotemporal patterns.