Magnetic separation is introduced to separate paramagnetic or superparamagnetic particles and beads from a suspension by implementing an external magnetic field gradient. The strength of this magnetic field declines with rising distance. Batch containers commonly separate nanoparticles from a fluid by imposing an external magnetic field gradient using a permanent magnet. In this system, magnetic particles aggregate the container's wall or bottom, therefore creating a continuous process without any aggregation is desired. In this paper, COMSOL Multiphysics ver. 5.6 (FEM) has been utilized to simulate a continuous microfluidic device with two geometrical designs, linear design (LD) and curved design (CD). The variables are the velocity of the primary fluid and buffer and the channel's width. The optimal designs of both microfluidic devices have been achieved by changing the variables in a specific range without aggregating them to the device's wall. The optimal structures for both designs were built and tested in the laboratory. The optimal velocity was 0.1 mm/s, the optimal channel width was 200 µm for LD, and it was 0.1 mm/s and 400 µm for CD, respectively. The separation percentage in the simulation was 63 %, the spectrophotometry result was 64.08 for LD, and CD showed better results in simulation (85 %) and laboratory experiment (80.24 %). So, this method can be used to optimize nanoparticle separation in microfluidic.