Since the Kriging model can provide the mean value of the performance function at a sample point and the corresponding variance among the various surrogate models, many Kriging-based reliability analysis methods have been developed to estimate accurate failure probabilities of engineering structure problems. Various active learning functions have been proposed to reduce the computation burden of Kriging-based reliability analysis for engineering problems. This paper proposes a new active learning Kriging-based method for estimating the failure probability. A penalty learning function is proposed to improve the efficiency and accuracy of the failure probability estimate based on a simple penalty function. Also, a distance constraint term is considered to avoid a cluster of the selected sample points. Additionally, by an error-based stopping criterion, convergence conditions of the proposed new active learning reliability analysis method are investigated. Finally, five numerical examples are provided to verify the accuracy and effectiveness of the proposed method and convergence strategy. Numerical results demonstrated that the novel proposed method significantly improves the computational efficiency of reliability analysis with high estimating accuracy of the failure probability.