To enable robust video transmission over heterogeneous networks, the hierarchical B-picture prediction structure is employed in the state-of-the-art video coding standard H.264/SVC, aiming to produce scalable bitstreams with various frame rates. However, the exhaustive mode decision process with the hierarchical B-picture structure increases the computational complexity of H.264/SVC encoding dramatically. In this paper, a fast mode decision algorithm is proposed to speed up H.264/SVC encoding with the hierarchical B-picture structure, which is achieved by utilizing macroblock (MB) features, correlation of temporal–spatial neighboring MBs, and the discrepant characteristics of hierarchical layers. Extensive experimental results demonstrate that the proposed algorithm is able to reduce the encoding time of H.264/SVC significantly for video sequences with a wide range of resolutions, and meanwhile the video quality and compression ratio are well preserved.