Full search has the best matching accuracy but costs the most time, while fast search algorithms can achieve a high speed but are easy to be trapped in local minimums. To compensate the shortcomings of the existing algorithms, this paper proposes a fast partial distortion motion estimation algorithm based on the multi-traps assumption (MT-PDS). It mainly consists of three steps: (1) estimate the number of traps in the search area, (2) obtain the positions of the traps by using the k-th (k=0,1,2,…15) partial distortion, which contributes most to the true sum of absolute difference (SAD), to perform the coarse search, and (3) get the positions of the deepest traps and search around them to get the global minimum. Besides, the proposed algorithm also introduces an adaptive search method and a sparse search pattern, which further reduce the computations. Experimental results show that the proposed MT-PDS is about 160 times faster than the full search on average; and the speed-up can achieve over 180 times for the low motion contents. What is more, it only degrades the quality by −0.0178dB and slightly increases the bit rate by 0.735%, which can be considered ignorable. Those advantages make the MT-PDS a very useful tool in real time applications, such as video compression, pattern recognition, target tracking, etc.