Graphene has been emerging as an ideal mid-infrared saturable absorber (SA) due to its broadband absorption, ultrafast nonlinear optical response, high stability and thermal tolerance. However, the current routes (e.g. chemical vapor deposition and spin coating) for constructing graphene SAs are suffering from the limited flexibility in substrate choice and the introduction of impurities during the transfer process, resulting in poor film quality and unstable laser modulation. Here, we demonstrate a high-quality graphene SA mirror (GSAM) grown directly on calcium fluoride (CaF2) substrate by a low-temperature plasma enhanced chemical vapor deposition (PECVD) method for mid-infrared pulse modulation. The controllable growth of high-quality graphene film on the nickel-modified CaF2 substrate is realized by adjusting the growth time and hydrocarbon ratio during PECVD process. Consequently, the GSAM shows excellent nonlinear optical absorption with the modulation depth of 11.2%. By inserting the GSAM into the Er:ZBLAN fiber laser, a stable passive Q-switched (QS) operation can be achieved with an average output power of 142 mW and a pulse width of 300.2 ns. The slope efficiency of QS laser is up to 17.4% and the peak power is 7.76 W. Our strategy paves the way for developing high quality and modulation stability GSAM towards industrial applications of pulsed mid-infrared lasers.