Robust and efficient indistinctive feature matching and outliers removal is an essential problem in many computer vision applications. In this paper we present a simple and fast algorithm named as LDGTH (Local Descriptor Generalized Hough Transform) to handle this problem. The main characteristics of the proposed method include: (1) A novel local descriptor generalized hough transform framework is presented in which the local geometric characteristics of invariant feature descriptors are fused together as a global constraint for feature correspondence verification. (2) Different from standard generalized hough transform, our approach greatly reduces the computational and storage requirements of parameter space through taking advantage of the invariant feature correspondences. (3) The proposed algorithm can be seamlessly embedded into the existing image matching framework, and significantly improve the image matching performance both in speed and robustness in challenge conditions. In the experiment we use both synthetic image data and real world data with high outliers ratio and severe changes in view point, scale, illumination, image blur, compression and noises to evaluate the proposed method, and the results demonstrate that our approach achieves achieves faster and better matching performance compared to the traditional algorithms.