Compared to surface plasmon resonance (SPR), the sensors based on the magneto-optical SPR (MOSPR) technique have much higher figure-of-merit (FOM). However, there are no reports about applying MOSPR in the optical fiber structure now. In this work, a novel D-shaped optical fiber sensor based on the MOSPR technique is proposed. The D-shaped optical fiber is coated with a thin silver film and a magneto-optical (MO) material film of Cerium-doped Yttrium-Iron garnet (CeYIG). By applying a magnetic field on the sensing region, the magneto-optical Kerr effect (MOKE) of the CeYIG layer and the related MOSPR phenomenon could be excited when appropriate light is transmitted in the proposed optical fiber sensor. The influence of the structural parameters including the residual cladding thickness, silver and MO material film thicknesses are analyzed theoretically by the finite element method (FEM). With the optimal parameters, the sensor achieves the sensitivity of 5304 nm RIU−1. Since the peak width of MOSPR spectra is much narrower than that of the SPR spectra, the FOM of the sensor is largely enhanced to 3864 RIU−1 on average and 13260 RIU−1 in maximum, which surpasses the optical fiber SPR sensors vastly. The miniaturized and simple design of the D-shaped optical fiber MOSPR sensor, coupled with the ultra-high FOM, offers itself great potential in biochemical sensing applications.