The performance of the phase shift keying (PSK) modulation technique over additive white Gaussian noise (AWGN) and multipath propagation channels generally becomes worse for higher-order modes. Therefore, a new modulation technique should be provided in order to have a system that is capable of transmitting data with higher efficiency while maintaining better performance at the same time. This paper presents the development of a fractal wavelet packet transform incorporated within the M-ary PSK system, namely M-ary PSK orthogonal wavelet division multiplexing (OWDM), which is proposed to obtain high performance of modulation in terms of spectrum efficiency and bandwidth resources intended for wireless communication systems. To demonstrate performance improvement over a Rayleigh frequency selective fading channel and in the presence of AWGN noise, the proposed system was evaluated and compared to the basic modulation system and M-ary PSK employing orthogonal frequency division multiplexing (OFDM). The performance results show that M-ary PSK OWDM had better performance in comparison with M-ary PSK OFDM and the conventional system. By utilizing 16 subcarriers, QPSK OWDM achieved bit error rate performance improvement from 1.5 × 10-3 to 1 × 10-4 for Eb/N0 of 20 dB with efficient bandwidth.