An increasing number of tunnels pass through the anomaly geothermal zones and inevitably encounter thermal engineering geological problems during tunnel operation. In order to efficiently eliminate the thermal damage, an efficient and energy-saving cooling system composed of borehole heat exchangers (BHEs) and thermal insulation layer in a high geotemperature tunnel is proposed. Taking Sangzhuling tunnel as an example, a 3D numerical model is established for analyzing the thermal performance of the cooling system, and is validated by the field measured data. The cooling effect of the cooling system in the high-geotemperature tunnels is proved to be remarkable in comparison with the tunnel having thermal insulation layer. Moreover, the effects of design parameters on the thermal performance are revealed during a 180-day heat extraction. The results show that a lower inlet temperature contributes to a shorter drop to the target tunnel ambient temperature. The effect of the cooling system is significantly improved as the borehole depth increases within a limited extent because the cooling effect cannot be infinitely improved by increasing the borehole depth. The thermal insulation layer plays a beneficial role in reducing the maximum ambient temperature and an adverse effect on the temperature reduction rate. These findings contribute to preliminary thermal performance estimation of the BHE cooling system in high-geotemperature tunnels without large-scale simulation.