Understanding the thermal conductivity (κ) of metal-organic frameworks (MOFs) with adsorbates is critical for tailoring their heat dissipation in practical applications. However, a detailed description of how adsorbates influence the heat transport of MOFs is still lacking. In this work, we employ molecular dynamics simulations combined with lattice dynamics calculations to investigate the effect of gas adsorbates on κ of representative crystalline and glassy zeolitic imidazolate frameworks (ZIFs). We identify that the diffusive adsorbate motions in the crystalline ZIFs with large pores can enhance lattice anharmonicity, serving as a source of phonon scattering and significantly suppressing κ. Conversely, the adsorbates with restricted dynamics in limited space (i.e., in the dense ZIF crystals or glasses) behave more like a heat transfer medium and facilitate heat transport. Besides pore size, gas dynamics are also proved to vary upon gas-lattice interaction strength, and dictate if gas adsorbates serve as a heat conduction suppressor or promoter. We anticipate that this governing role of gas dynamics is applicable to a diverse range of MOF-adsorbate systems. As such, our work offers guidance for the practical thermal management of MOF porous materials to ensure their desired storage or separation performance.