In response to the possibility of time delay (TD) signatures (TDSs) leakage becoming a serious problem in chaotic optical communication, from the perspective of improving the security of chaotic optical communication, this paper proposes a chaotic secure optical communication system based on dispersion keying, which generates signals with TDSs concealment characteristics. In this scheme, a common laser is used to drive chaotic synchronization between the transmitter and the receiver. The complex chaotic signal is generated by the combination of all-optical feedback and optical injection, and the TDSs are hidden by using the structure of dual filters and setting the parameters of each device reasonably. The switch is controlled by binary message to connect fiber Bragg gratings (FBGs) with different dispersion coefficients and maps different symbols to different chaotic attractors instead of the traditional way of directly sending messages to the channel. In addition, the generated signal no longer suffered from return map attacks. FBGs can compress and expand signals in the time domain to enhance the complexity of chaotic signals, and the dispersion value is an important key to effectively increasing the key space. Finally, the performance of TDSs concealment, sensitivity of parameter mismatch, and return map attack in this scheme are discussed in detail. The scheme still achieves a bit error rate (BER) of 1.74 × 10−4 when transmitting at a rate of 10Gbit/s under harsh communication environments such as 10 dB signal-to-noise ratio (SNR), which provides a reference for realizing high-speed and secure optical communication at the physical layer.