M-type hexagonal ferrites have been getting considerable attention owing to their promising application in electronic fields. Though, the growth of nanosized M-type hexagonal ferrites is still a big challenge. Herein the fabrication of M-type hexaferrite with nominal composition Ca1-xBixFe12-xCoxO19 (x=0.00, 0.05, 0.10, 0.15, 0.20) with high quality is reported by the sol-gel auto combustion route. The objective of the study is to improve the structural, spectral, dielectric, and magnetic characteristics of M-type hexagonal ferrite which was achieved through the variation of concentration of Cobalt and Bismuth. X-ray diffraction (XRD) patterns confirmed the single-phase M-type hexagonal structure. The crystallite size of all samples was found to be in the range of 42–49 nm. Other parameters such as lattice parameters a & c, unit cell volume, crystallite size, X-ray density, Bulk density, and porosity were also calculated. The doping contents were found to decrease the bulk and X-ray densities while increasing the porosity. Fourier-transform infrared (FTIR) spectra showed the formation of metal-oxygen stretching vibrations that confirmed the formation of hexagonal ferrites. The scanning electron microscopy (SEM) images revealed a regular platelet hexagonal structure and homogeneously distributed grains were examined. The dielectric constant was high at low frequency and then decreased with increasing frequency, while the dielectric loss was decreased appreciably with doping. The saturation magnetization ranged from 15.51 to 38.27 emu/g, coercivity increased from 207.93 to 1359.69 Oe, and the squareness ratio was found to be in the range of 0.19–0.78. The dielectric and magnetic properties of Ca1-xBixFe12-xCoxO19 (x=0.00, 0.05, 0.10, 0.15, 0.20) with the variation of Co and Bi revealed that these materials are good candidates for modern devices.