The high hardness and polycrystalline structure of polycrystalline magnesium aluminate spinel (PMAS) present significant challenges in achieving high-precision processing, particularly during polishing. Issues such as low processing efficiency and grain effects can compromise processing accuracy, surface quality, and optical performance. To address these challenges and enhance both polishing efficiency and surface quality, this study employed agglomerated diamond (AD) abrasives and single crystal diamond (SCD) abrasives in force rheological polishing (FRP). The experimental results revealed that the lowest surface roughness (Sa) and optimal selective material removal were achieved using 25 μm AD abrasives, particularly those agglomerated with 0.2 μm SCD abrasives. This approach reduced Sa from 146.1 nm to 4.6 nm, peak-to-valley height difference (Sz) from 1329 nm to 69.3 nm and achieved a material removal rate (MRR) of 228.8 nm/min. Conversely, polishing with 1 μm SCD abrasives resulted in poorer selective material removal, with Sz decreasing from 1265.1 nm to 280.9 nm and Sa decreasing from 138.3 nm to 27.6 nm, although it exhibited the highest MRR of 323.64 nm/min. The superior polishing results obtained with AD abrasives, compared to SCD abrasives of the same size, can be attributed to the polycrystalline-like structure of AD abrasives, which exhibit more micro cutting edges. Additionally, the surface and subsurface damage after polishing with AD abrasives was analyzed. The results indicated that material removal of PMAS using AD abrasives was effective in FRP, resulting in minimal lattice distortion. This study provides both theoretical and experimental foundations for polishing PMAS using larger-sized AD abrasives in FRP.