Membrane technology can realize wastewater recycling in industry, but its efficiency is threatened by fouling. In this study, magnetic palygorskite (m-Pal) nanofibers were obtained by anchoring Fe3O4 nanoparticles onto the surface of palygorskite. Then, composite membranes composed of poly(vinylidene fluoride) (PVDF) and m-Pal were prepared during a magnetic field assisted phase inversion process. The m-Pal can be elevated to the membrane surface due to magnetic force, thus adjusting membrane performance. The PVDF/m-Pal composite membrane prepared with magnetic field displayed higher water flux and better hydrophilicity than that of pristine PVDF membrane. The water flux can be increased to 343.2 L·m−2·h−1·bar−1, and water contact angle decreases to 71.8° with the addition of 7% m-Pal. Moreover, the anti-fouling and easy-cleaning performance of the composite membrane was demonstrated in bovine serum albumin (BSA) solution separation, with BSA flux and flux recovery ratio increased by 41.3% and 155.4%, respectively, compared with the control PVDF membrane.