A smart insulation (SI) winding technique utilizing vanadium trioxide (V2O3) as a turn-to-turn insulation material is an effective means of achieving a suitable time constant and thermal stability in high-temperature superconducting (HTS) coils. However, the low electrical conductivity of the V2O3 makes the SI coil vulnerable to damage during quenching. To address this problem, this study explored a mixture of V2O3 and molybdenum (Mo) particles to improve the thermal stability of the SI coil. Two GdBCO coils, insulated with V2O3 and V2O3:Mo, respectively, were fabricated, and quench tests were performed at various operating currents in liquid nitrogen at 77 K. The findings revealed that the V2O3:Mo-insulated coil demonstrated improvements of 82%, 25%, and 160% in the minimum quench energy, and the normal-zone propagation velocities in the longitudinal and transverse directions, respectively, at 60% of the current-carrying capacity of the coil, which are essential performance parameters in the operation of a superconducting magnet.