The H.264/AVC standard achieves higher coding efficiency than previous video coding standards with the rate-distortion optimization (RDO) technique which selects the best coding mode and reference frame for each macroblock. As a result, the complexity of the encoder have been significantly increased. In this paper, a fast intra-mode decision algorithm is proposed to reduce the computational load of intra-mode search, which is based on the inverse tree-structure edge prediction algorithm. First, we obtained the dominant edge for each block from local edge information, then the RDO process is only performed by the mode which corresponds to dominant edge direction. Then, for the (or ) block stage, the dominant edge is calculated from its four (or ) blocks` dominant edges without additional calculation and the RDO process is also performed by the mode which is related to dominant edge direction. Experimental results show that proposed scheme can significantly improve the speed of the intra prediction with a negligible loss in the peak signal to noise ratio (PSNR) and a little increase of bits.