Multiple-input multiple-output (MIMO) technology can provide high data rate and substantial diversity. Space time block code (STBC) provides diversity gain and obtains signal of better quality even under bad channel conditions. However, it has no contribution to system average throughput. Vertical-bell laboratory layered space-time (VBLAST) improves peak data rate but leads to higher packet error rate (PER) at cell edge when the channel condition is poor. To get both higher system average throughput and cell edge throughput, an adaptive STBC-VBLAST transmission for downlink mobile WiMAX system is proposed and analyzed through system-level simulation in this article. The proposed approach compares the effective capacity of two candidate MIMO modes (STBC or VBLAST) and reports its decision to the base station with only one extra signaling bit. The simulation results show that the proposed approach can attain a balance between STBC and VBLAST. Besides, the cell throughput approximates that of pure VBLAST and the cell edge throughput is close to that of pure STBC.