Calcium aluminate cement (CAC) is extensively applied as castable binder, and the drying process is a crucial step before castables come into service. Limited researches have focused on the hydration and phase transformation of CAC in the castables during the evaporation stage of drying process, in which the free water is gradually discharged. This work involves the investigation of the strength development of CAC-bonded alumina castables and phase evolution of the hydrates during drying at 110 °C. After curing for 24 h at 30 °C, the CAC-bonded alumina castables and matrices were dried at 110 °C for different periods of time to explore the effect of drying time on the mechanical property of CAC-bonded castables and on the phase assemblages and microstructure of the castable matrices. The results demonstrate that the initial drying period prompts the rapid conversion from metastable C2AH8 to stable C3AH6 as well as the continued hydration of the residual CA and CA2. The drying strength of the castables is significantly enhanced in comparison with demolding strength.