The accurate reconstruction distance is one of the important parameters in digital holography. Typical autofocusing methods have been proposed for digital holography to determine the focusing plane. In these methods, a series of holograms at fixed-step intervals always needed to be reconstructed and then locate the focal plane by using an evaluation function, a procedure that is very time-consuming. In this paper, an accurate and fast autofocusing method for off-axis digital holograms is proposed. The method first employs the step reduction search algorithm and the integral amplitude modulus (AMP) evaluation function with low computational effort to iteratively reduce the searching range of the focal plane. Then the particle swarm optimization (PSO) algorithm and a difference-in-amplitude (DIF) evaluation function with high sensitivity are used to accurately locate the focal plane. Numerical and experimental results show that the proposed method significantly improves the focusing plane detection accuracy and computational efficiency compared with the typical autofocusing methods.