Previous structural neuroimaging studies linked cerebellar deficits to temporal lobe epilepsy (TLE). The functions of various cerebellar regions are increasingly being valued, and their changes in TLE patients warrant further in-depth investigation. In this study, we used the Spatially Unbiased Infratentorial (SUIT) toolbox with a new template to evaluate the cerebellar structural abnormalities in patients with TLE, and further explored the relationship between the changes of different cerebellar regions and cognition. Thirty-two patients with TLE were compared with 39 healthy controls (HC) matched according to age, gender, handedness, and education level. All participants underwent a high-resolution T1-weighted MRI scan on a 3.0 Tesla scanner. We used a voxel-based morphometry (VBM) approach utilizing the SUIT toolbox to provide an optimized and fine-grained exploration of cerebellar structural alterations associated with TLE. Compared with HC, TLE patients showed a significant reduction in the volume of gray matter in the Left lobule VI and white matter in the Right Crus II. In the TLE patient group, we conducted partial correlation analysis between the volumes of different cerebellar regions and cognitive rating scale scores, such as MMSE and MoCA. The volume of the Left lobule VI (GM) exhibited a positive correlation with the MMSE score, but no significant correlation was found with the MoCA score. On the other hand, there was no significant correlation observed between the volume of the Right Crus II (WM) and the two cognitive scale scores mentioned above. Furthermore, it was observed that the MMSE was more effective than the MoCA in identifying epilepsy patients with cognitive impairment. This study supported previous research indicating that temporal lobe epilepsy (TLE) is linked to structural changes in the cerebellum, specifically affecting the volume of both gray and white matter. These findings offer valuable insights into the neurobiology of TLE and hold potential to inform the development of enhanced diagnostic methods and more effective treatment approaches.