The development of magnetic separation technology using magnetic nanoparticles offers a promising avenue for targeted drug delivery and dealing with the upcoming water crises, environmental pollution and the gradual mineral resource depletion. In this study, a three-dimensional Lagrangian Discrete Phase Model (DPM) is carried out to simulate the performance of Fe3O4 nanoparticles to improve the separation process under the influence of an external magnetic field within a horizontal pipe. The crucial role of the drag force in capture efficiency (CE) prompted its examination, simulating various drag models for groups of particles. The Stokes-Cunningham model, showing a 3.59% average error is a suitable choice compared to experimental results. The research examines the impact of effective parameters, including flow velocity, magnetic field intensity, wire location, particle size and mass flow rate, and pipe diameter on CE and flow pattern. The results show that increasing nanoparticle concentration reshapes the flow pattern due to secondary flows without significantly changing separation efficiency. Moreover, decreasing flow velocity, diminishes drag force and enhances magnetic force impact. Specifically, reducing the velocity to a third increases CE by 37%. Furthermore, capture capacity varies approximately linearly with electric current. Due to the magnetic force’s role as a volumetric force in interphase momentum transfer, the increase in particle size from 200 to 500 nm at 3 × 105 A enhances CE by nearly 50%. However, increasing the pipe diameter diminishes particle capture, attributed to higher Reynolds numbers. According to the results, the impact of increasing magnetic field intensity and particle size on CE improvement is notably more pronounced compared to the effect of flow velocity reduction. A comparative analysis of three injection types reveals that using the group injection type helps to select an appropriate injection location to increase CE and identify the final positions of nanoparticles.