A novel low-loss Ba12SbNb9O36 microwave dielectric ceramic is synthesized via the solid-state reaction method. TG-DSC thermal analysis is employed to study the phase formation process of the samples. The XRD and TEM analyses reveal that the ceramic samples all confirmed a hexagonal perovskite structure with the R-3 m space group. The refinement result shows that the ceramic is a shifted hexagonal perovskite with an 18-layer structure. Porosity has an important impact on permittivity and dielectric loss in this ceramic. Raman spectroscopy revealed that the quality factor shows a close connection with the FWHM of the A1g mode at 777 cm−1. XPS analysis shows that the oxidation of Sb ions deteriorates the τf value of the ceramic. The optimum microwave dielectric properties (εr = 35.59, Q × f = 58,297 GHz and τf = +34.14 ppm/°C) are obtained by sintering at 1250 °C, making the Ba12SbNb9O36 ceramic a promising candidate for high-performance microwave applications.