Polyvinylchloride (PVC) and acrylonitrile-butadiene-styrene (ABS), as common components in waste plastic, are hard to separate due to similar physical and chemical properties, which limits plastic recycling and cleaner production. In this research, a novel approach by using peroxymonosulfate (PMS) was established to modify waste PVC and ABS surface, and separation was successfully realized in flotation process. Key variables closely related to PMS oxidation capacity, namely pH, temperature, and concentration were confirmed through single parameter experiments. Subsequently, response surface methodology (RSM) using Box-Behnken design (BBD) was applied to optimize the above surface treatment conditions. A quadratic model was generated for predicting PVC purity (response value). Mechanism of surface modification was explored by Scanning electron microscope (SEM), Fourier transform infrared spectroscopy (FT-IR), X-ray photoelectron spectroscopy (XPS). Results showed that the optimal conditions were PMS concentration of 0.026 mol/L, pH value of 9.0, and temperature of 65 °C, respectively. PVC can achieve 99.72% purity and 100.00% recovery, and ABS can reach 100.00% purity and 99.81% recovery in triplicate validation tests, respectively. Additionally, available ranges of particle sizes, mass ratios, and reuse of reagents further show the remarkable application potential of flotation combined with PMS treatment in separating waste PVC and ABS. The mechanism of surface treatment shows that oxygen-containing groups are introduced into ABS surface, enhancing the hydrophilicity of ABS, but PVC is almost unaffected and remains floating in flotation process. This study established a novel surface modification with RSM optimization and reliable mechanisms for the flotation separation of waste PVC and ABS.