A hybrid technique in optical wireless communications is an effective solution to counter impairments and to improve link availability. In this work, a hybrid frequency phase keying (FPK) modulation scheme, based on frequency shift keying (FSK) and phase shift keying (PSK), for a hybrid radio frequency (RF)/free space optical (FSO) communication system is proposed. The hybrid FPK modulation scheme combines advantageous features of FSK and PSK and has the possibility to provide improved link performance in adverse channel conditions. A transmit aperture selection (TAS) and maximal ratio combining (MRC) diversity-based multiple-input–multiple-output (MIMO) hybrid RF/FSO system is considered to mitigate adverse fading effects. Analytical closed-form expressions are derived for the average symbol error probability to trace the performance of the TAS-MRC-based MIMO hybrid RF/FSO system with FPK modulation over fading and pointing error conditions. A comparative analysis of average symbol error performance is presented. The results clearly indicate that the proposed FPK modulation scheme with the MIMO hybrid RF/FSO communication system link is advantageous, as it exploits the complementary properties of both RF and FSO channel characteristics and PSK and FSK modulation schemes. In addition, the proposed MIMO hybrid RF/FSO system with FPK modulation offers superior performance, compared with MIMO RF, FSO, or hybrid RF/FSO systems applied with FSK and PSK modulation schemes. An optimum beam waist is derived analytically, and it is further verified with a numerical optimization technique. All results are validated using Monte Carlo simulation.