This paper proposes a high-security multidimensional data protection system based on the Hartley algorithm-driven chaotic scheme. We utilize the fast Hartley algorithm instead of the fast fourier computation, and we employ chaotic sequences generated by the multi-winged chaotic system to achieve chaos-driven 3D constellation mapping, effectively integrating the chaotic system with the stochastic amplitude modulator. We reduce the signal’s peak-to-average power ratio (PAPR) by deploying a random amplitude modulator. Simultaneously, this approach enhances the security of the physical layer of the signal. The PAPR reduction can reach up to 2.6 dB, while the most robust and stable modulator scheme can gain 2 dB. Finally, in the Hartley frequency domain, the signal's frequency is disrupted, providing the entire system with a key space of 10131 to resist violent cracking and thus improving the system's overall security. To validate the feasibility of our scheme in comparison to conventional IFFT-based encrypted 3D orthogonal frequency division multiplexing, We achieved a transmission rate of 27.94 Gb/s over a 2 km multicore fiber. Experimental results show that since the random amplitude generator effectively reduces PAPR, our proposed encryption scheme increases the forward error correction threshold range by 1.1 dB, verifying that our proposed scheme has highly reliable security performance.