The iterative closest point (ICP) algorithm is an efficient method to register point sets which may fail as the rotation is various. To improve the robustness of registration and reduce the variety of rotation, the boundary of the rotation angle is introduced into the 3D point set registration problem in this paper, which is described as a least square registration model with inequality constraints. The new problem is solved by a more robust ICP approach with the bounded rotation angle which repeats two steps. Firstly, the correspondence between two point sets is set up according to the known rigid transformation. Secondly, to compute the rotation angle of the objective function with boundary, a closed-form solution of the transformation is obtained according to the monotonic property of the objective function in the given interval. The proposed algorithm is demonstrated to monotonically converge to a local minimum from any given initial value. Therefore, to obtain the desired results, the boundary of rotation angle and initial value are estimated by the principle component analysis. A series of experiments are conducted to demonstrate that the proposed method is much more robust without increasing the computational complexity compared with the state-of-the-art point set registration method. This paper proposes a robust 3D registration method by introducing a rotation angle with boundary, and it can be solved with a closed-form solution for the transformation at each iterative step.This paper proposes a boundary and initial value estimation method, which relieve the local minimum problem effectively.The proposed method is successfully applied on 3D point sets, which contain outliers and noise, and the captured face point sets with high accuracy.