The heat dissipation of high alumina castables for ladle permanent layer is a topic worthy of attention and solution. It is proposed to use spherical porous mullite (a solid waste) instead of irregular M70 mullite aggregate in castable to reduce the thermal conductivity and realize a high-strength and lightweight design. The effect of spherical porous mullite content in castables on thermal conductivity, service strength and thermal shock resistance were discussed. The results demonstrated that the substitution amount of spherical porous mullite increased, the bulk densities of the castables dropped from 2.68 to 2.38 g.cm−3, the apparent porosity rose from 15.5 to 25.0%. The thermal conductivities reduced from 1.174 to 0.585 W.m−1.K−1 (1000°C), which can be ascribed to the increased porosity extending the heat transfer path. Furthermore, owing to the high strength of spherical mullite and its excellent bonding with matrix, the mechanical strength of castable was at a high strength level, with modulus of rupture of 18 M Pa and cold crushing strength of 76 MPa at the spherical porous mullite completely substituting M 70 mullite. The semi-lightweight and low thermal conductivity castable containing spherical porous mullite show great application potential in the ladle permanent layer.