For fully utilizing the system frequency response of optical intensity-modulation and direct-detection (IM/DD) passive optical network (PON) transmission system, this paper demonstrates a novel adaptive quadrature amplitude modulation (QAM) encoder /decoder for bit and power loading based on adaptively modulated orthogonal frequency division multiplexing (AM-OFDM). The applied adaptive QAM encoder/decoder supports all the M-QAM formats on each sub-carrier instead by ranging the order of modulation format on different sub-carriers. Simulation results show that the adaptive QAM coder/decoder can be used for variable transmission channel with different transmission distances between optical line terminal (OLT) and optical network units (ONUs). Meanwhile, the adaptive loading algorithms minimizes the bit error rate (BER) for a given bit rate. In addition, compared to the fixed 16QAM scheme, the receiver sensitivity performance of the adaptive QAM scheme is improved by 5.2 dB for 30 km standard single-mode fibre (SSMF) transmission at the -8.12 dBm received optical power (ROP).