The spontaneous combustion of coal in goaf was effectively prevented by adding nano-aluminum hydroxide (Nano-ATH) into the composite foaming agent solution in the preparation of a nano-ATH foam with favorable formability and high stability. The compound scheme and amounts of surfactants and nano-ATH were studied using an interfacial rheometer and foam scanner, and the flame retardant property of the prepared nano-ATH foam was explored with a cone calorimeter. Results showed that when surfactants B and E were compounded in a mass ratio of 1:1. The compounded system had a mass concentration of 0.4% and the most uniform and finest foam. After nano-ATH was added, the viscoelasticity modulus of the foam liquid film was elevated from 0.91 mN/m to 3.08 mN/m, and the foam volume increased from 380 mL to 700 mL. At a mass concentration of 2%, the half-life period of the foam increased from 78 s to 453 s, indicating that the nano-ATH increased the mechanical strength of the liquid film, enhanced foam stability, and enlarged the foaming volume. In the coal samples with nano-ATH foam, the time to ignition increased from 26 s to 176 s, time of sustained combustion decreased from 520 s to 211 s, peak heat release rate declined from 117 kW/m2 to 58 kW/m2, and peak smoke production rate declined from 0.084 m2/s to 0.0049 m2/s during combustion, all indicating good flame retardant and smoke-suppression properties.