DOE with wide diffraction angle requires the feature size down to subwavelength. However, the emerging vectorial effect of light beam makes the scalar diffraction theory invalid. To address this problem, a hybrid iterative design method is proposed for 2D beam splitter in this work. Firstly, several initial solutions are obtained by non-paraxial scalar iterative Fourier transform algorithm (IFTA). Then, rigorous coupled wave analysis (RCWA) is used as evaluation function to compute the diffraction efficiency and uniformity. Combined with the Genetic Algorithm, the second optimization of DOE is implemented to obtain the final optimal surface relief structure. Numerical simulations and optical experiments are both performed to verify the proposed algorithm. The results obtained by the experiments and the simulation results are compared, and the relevance between the two results are above 80%. This work can open a new way for the 2D DOE design with wide diffraction angle.