Because the M estimation method may lead to poor robustness or even failure owing to excessive outliers, a robust algorithm with a high breakdown point was proposed and applied to the estimation of coordinate transformation parameters. Firstly, the sampling method was used to calculate multiple sets of model parameters, and some sampling results were sifted according to posterior information. Then, the samples were sorted according to their number in the sampling results, and the F-test was adopted to screen and reserve valid information. Finally, the initial values of the reliable parameters were computed using the valid information, and the final parameters were obtained by the Institute of Geodesy and Geophysics III scheme. Monte Carlo method was adopted for the simulation test, and a case analysis was chosen for verification. The results show that the proposed method can identify and process outliers more accurately than those of Rousseeuw and Hubert (2011 Wires Data Min. Knowl. 1 73–79) and Tao et al (2016 Acta Geod. Cartogr. Sin. 45 297–301). When the proportion of outliers exceeded 50%, the proposed algorithm maintained a strong robustness and had a high breakdown point.