High-energy photons of gamma-ray bursts (GRBs) might be emitted at different intrinsic times with energy dependence at the source. In this letter, we expand the model from previous works on testing the Lorentz Invariance Violation (LV) with the observed GRB data from the Fermi Gamma-ray Space Telescope. We reanalyze the previous data with the full Bayesian parameter estimation method and get consistent results by assuming that the time delays are due to an LV term and a constant intrinsic time delay term. Subsequently, we neglect the LV effect and only consider the intrinsic time delay effect. We assume a common intrinsic time delay term along with a source energy correlated time delay of high-energy photons. We find that the energy-dependent emission times can also explain the observed GRB data of high-energy photon events. Finally, we integrate these two physical mechanisms into a unified model to distinguish and evaluate their respective contributions using the observed GRB data.