The introduction of probability model into point set matching has achieved great success. Because the image point set has more outliers and the image has complex transformation, the traditional Gaussian Mixture Model (GMM) does not perform well in image point set matching. In this paper, we propose a Phased Gaussian Mixture Model (PGMM) for point set registration between images. Different from the existing registration algorithms based on GMM, PGMM uses Locality Sensitive Hashing to distinguish the feature reliability of feature points. The initial transformation relation is obtained in the segment with high feature reliability and vector constraint is applied to other segments. The average Hamming distance is used to give different weights to each Gaussian component, which is related to the characteristics between data points and model points. The proposed method has good performance in the case of large image transformation scale.