This paper jointly considers the mechanisms of rate control and the rate-distortion optimization (RDO) for the H.264/AVC hardware encoder. The proposed architecture performs intra prediction and inter prediction with our rate control mechanism in the first two stages. In addition, a low complexity macroblock (MB)-level mean absolute difference (MAD) prediction algorithm which considers both the motion information and the MAD value in the neighboring and current MBs is proposed. The proposed rate control algorithm can reduce the computational complexity and does not suffer from data dependency problems which may decrease the rate-distortion performance, thus it is suitable for H.264/AVC hardware encoders. Furthermore, the RDO-based architecture for the H.264/AVC encoder reduces the number of accurate rate-distortion cost calculators while maintaining the high rate-distortion performance. Experimental results show that the proposed rate control algorithm with the RDO-based architecture can improve the rate-distortion performance of H.264/AVC.