Fe3O4-octadecyltrichlorosilane (Fe3O4-OTS) was synthesized and used to remove dyes in a competitive system. Fe3O4-OTS was prepared by slow hydrolysis of OTS in cyclohexane on the surface of Fe3O4 obtained through coprecipitation method. Scanning electron microscope (SEM), energy dispersive spectrometer (EDS), and contact angle analyzer (CA) were used to analyze the properties of Fe3O4-OTS. Methyl orange (MO) and methylene blue (MB) were selected as model molecules to study the influence mechanism of pH and ionic strength on competitive adsorption. The results of EDS and CA indicated that Fe3O4 was modified successfully with OTS on the surface. Silicon appeared and carbon content increased obviously on the surface of adsorbent. Contact angle of adsorbent increased from 0° to 107° after being modified by OTS. Fe3O4-OTS showed good separation for MO and MB in competitive system, which has potential to separate dyes in sewage. Separation factor (βBO) changed from 18.724 to 0.017, when pH changed from 7 to 12, revealing that MO and MB could be separated almost thoroughly by Fe3O4-OTS. pH could change the surface charge of Fe3O4-OTS and structure of dyes, and thus change the interactions of competitive system indirectly. Even though hydrophobic interaction was enhanced, ionic strength reduced the difference of electrostatic interaction between dyes and Fe3O4-OTS. So it is unfavorable to separate dyes with opposite charges when ionic strength increases. These findings may provide theoretical guidances to separate two-component dye pollutants.