AbstractThis paper proposes a new type of hybrid eighth‐mode substrate integrated waveguide (EMSIW) and microstrip line (MSL) bandpass filter (BPF) with mixed electric and magnetic coupling. Firstly, two second‐order EMSIW BPFs with mixed coupling and wide stopband performance are investigated. Then, in order to realise high‐order BPF with decent performance, half‐wavelength (λ/2) open‐circuited MSL resonators are employed in the hybrid BPFs design. In the end, two third‐order BPFs and a fourth‐order BPF are developed by combining the EMSIW and MSL resonators together. Extra transmission zeros can be introduced due to the cross couplings in the trisection and quadruplet topologies. For demonstration, the proposed prototypes are designed, fabricated and measured. Reasonable agreements can be achieved between the measurement and simulation. The proposed third‐order BPFs compete against the state of the art, exhibiting the competitive characteristics of the operating bandwidth, physical footprint and upper‐stopband rejection with less order. The proposed fourth‐order BPF features a compact size of 0.34 × 0.34λ0, a low insertion loss of 1 dB, a broad 3‐dB fractional bandwidth of 22.9% and an excellent lower‐stopband suppression.