Abstract This paper proposes and demonstrates a novel dynamic quadrature amplitude modulation (QAM) mapping scheme to provide a secure physical-layer transmission in orthogonal frequency division multiplexing passive optical network (OFDM-PON). In the proposed scheme, permutated QAM symbols are independently and randomly mapped onto chaotically selected new locations on the complex plane. With the proposed encryption scheme these new locations are within the dimensions of a conventional QAM. A hyper digital chaos is applied to perform a multi-fold encryption in the proposed scheme which involves dynamic QAM mapping as well as time domain and frequency domain permutations on QAM symbols. As a result, a noise-like constellation with a key space of ∼10310 is achieved to enhance the high-level security in physical-layer transmission. A successful transmission experiment of 9.4-Gb/s encrypted 16-QAM OFDM signals is demonstrated over 25-km standard single mode fiber (SSMF), where the robust security and transmission performances are validated.