The photovoltaic/thermal systems with magnetic nanofluid based spectral beam filter would achieve thermal decoupling and utilize the full spectrum to minimize energy loss. The prepared Fe3O4 magnetic nanofluid was introduced into this system as selective absorbing fluid filter. An external magnetic field was added to manipulate the distribution behavior of nanoparticles that affect the optical characteristics of magnetic nanofluids could meet the various application requirements for electrical/thermal output of the photovoltaic/thermal system. The optical properties of magnetic nanofluid with various nanoparticle distribution behaviors under the varying magnetic field were tested by UV–visible spectrophotometric and Finite-Difference Time-Domain simulated respectively. Furthermore, the effect of magnetic nanoparticle concentration and magnetic field strength on the electrical/thermal output performance of the photovoltaic/thermal system was studied. The results show the higher nanoparticle concentration enhances the thermal output of the system but reduces the electrical output, accompanied by a significant reduction in the surface temperature of the cell. Both photothermal and photovoltaic conversion efficiencies are increased below 100mT magnetic field due to the transmission in the response spectrum of monocrystalline silicon cell increase under the formation of chain structure of the nanoparticles. It is concluded that the photothermal, photovoltaic, and total solar energy conversion efficiencies are respectively 59.82%, 13.98%, and 73.75% at 0.0025 wt% concentration under 100 mT magnetic fields, and its function of merit value was increased by 109% compared to the system without spectral beam filter.