Based on the recent experiments on the hybrid quantum system of a superconducting microwave cavity coupling strongly to an inhomogeneous broadening spin ensemble under an external driving field, we use the exact master equation approach to investigate its non-Markovian decoherence dynamics. Here the spin ensemble is made by negatively charged nitrogen-vacancy (NV) defects in diamond. Our exact master equation theory for open systems depicts the experimental decoherence results and reveals the mechanism how the decoherence induced by the inhomogeneous broadening is suppressed in the strong-coupling regime. Moreover, we show how the spectral hole burning generates localized states to further suppress the cavity decoherence. We also investigate the two-time correlations in this system to further show how quantum fluctuations manifest quantum memory.