The comprehensive properties and sintering temperature of microwave dielectric ceramics are the key factors limiting their practical application. In this work, a novel strategy of complex (Co1/3Nb2/3)4+ ions substituted and adding glass (Li2O-B2O3- SiO2-CaO-Al2O3: LBSCA) was proposed to greatly enhance the microwave dielectric properties of Ba3Ti4-x(Co1/3Nb2/3)xNb4O21 (0 ≤ x ≤ 12 mol%, xCN-BTN) ceramics. A new ceramic of (Co1/3Nb2/3)4+ substitution improved hexagonal phase BTN was successfully fabricated and systematically investigated. As the increased of (Co1/3Nb2/3)4+ ions concentration, the Q×f was greatly increased because of higher packing fraction, and the reduction in the [Nb1/Ti1O6] octahedral distortion caused great changes of τf. Besides, the P-V-L theory revealed that the Ti/Nb ratio greatly contributed in permittivity εr, and the Nb-O bonds mainly affected dielectric loss, especially in Nb(1)-O2(1) bond. As a result, the superior microwave dielectric properties were obtained in 6CN-BTN ceramic sintered at 950 °C: εr∼40.99 ± 0.48, Q×f∼17,310 ± 565 GHz, and τf∼8.62 ± 4.34 ppm/°C. More encouragingly, 6CN-BTN showed good compatibility with silver, making it a promising K40 LTCC material for filter and dielectric resonators. This study presented a new strategy to improve the properties of BTN-based materials by complex ions substitution, which was expected to provide a general approach for the design of advanced microwave dielectric ceramics for other material systems.