The rare earth elements (REEs) industrial wastewater is characterized by high ammonium nitrogen and low-strength organic compounds. Reverse osmosis (RO) process is effective for the REEs wastewater treatment. However, membrane fouling deteriorated the RO process. In this work, the fouling mechanism during RO process of REEs wastewater was elucidated via multiscale methods. A series of bench-scale fouling tests with simulated REEs wastewater containing high NH4+–N and different concentrations of 2-ethylhexyl phosphonic acid mono-(2-ethylhexyl) ester (P507) were performed with a commercial RO membrane to evaluate the fouling extent of the RO process. A critical P507 concentration (0.25 mg L−1) was observed where the fouling pattern changed qualitatively. When the P507 concentration was lower than 0.25 mg L−1, the relative flux increased and the membrane surface became more hydrophilic. When P507 reached this critical point, severe fouling occurred accompanied with a more hydrophobic membrane surface. Multiscale simulations [i.e., molecular dynamics (MD) and dissipative particle dynamics (DPD)] revealed that the fouling layer network varied with P507 concentration. This work provides in-depth insights into membrane fouling mechanism in the REEs wastewater, and has enlightening significance for fouling control strategies and the innovation of anti-fouling membrane materials.