Quasi-Zero Stiffness (QZS) is essential for low-frequency passive vibration isolation, with the negative stiffness mechanism (NSM) being pivotal for achieving QZS. Mechanical spring-based NSMs often face issues like contact friction, parasitic high-frequency modes, and stiffness nonlinearity. This paper presents a new magnetic NSM designed to overcome these challenges effectively. The mechanism is applied to a Fast Steering Mirror (FSM) as a case study, enhancing its vibration isolation performance. Initially, a torque model for the magnetic NSM is established, exploring the impact of various structural dimensions on torque and nonlinear issues. The magnetic NSM features adjustable negative stiffness and a large, adjustable linear working range. Experiments were designed to validate the principle of the magnetic NSM and its effect on the QZS FSM’s vibration isolation performance. The experimental results confirm the accuracy of the magnetic NSM principle. The QZS FSM stiffness is significantly reduced to quasi-zero stiffness, resulting in a decrease of the resonance peak from 20.1 to −12.8 dB and lowering the minimum suppressible vibration frequency from 54.63 to 3.44 Hz, thereby enhancing the vibration isolation performance of the QZS FSM.