—In the work, we propose a new chaotic encryption algorithm for optical physical layer security enhanced in coherent optical orthogonal frequency division multiplexing (CO-OFDM) systems. The proposed algorithm adopts the six-dimensional (6D) hyperchaotic to achieve the generation of key sequences, which can effectively improve the unpredictability and security of the system. The encryption scheme includes two steps, first, the binary sequence is disturbed by DNA and RNA two-stage encoding and RNA mutation, and then the initial encrypted data is injected into the algorithm module with two-dimensional continuous chaotic mapping (2D-SCL) and Brownian motion to scramble successively. Simulations in a 16QAM CO-OFDM system at a symbol rate of 30 Gbaud over 80 km of standard single-mode fiber (SSMF) show that a legitimate user is able to successfully decrypt the received signals, and an illegal eavesdropper obtains a useless signal with a bit error rate (BER) of about 0.5. The scheme can tolerate very small deviations and has a key space of approximately 10294, which can effectively withstand illegal attacks. And the peak-to-average power ratio (PAPR) can be reduced by about 1.01 dB.