The high accuracy, low drift low-level radio frequency (LLRF) system is essential for the long-term stability of the accelerator RF and the acquirement of low emittance, high intensity electron beams. A time-multiplexing pick-up/reference signal based LLRF system is proposed to deal with the component temperature related phase drift and has been deployed and applied at the Xi'an Gamma-ray Light Source (XGLS) injector. The long term dual-receiver out-of-loop stability experiments with a continuous wave laser based phase reference distribution system (PRDS) show that the LLRF system can achieve ∼40 fs Root-Mean-Square (rms) phase accuracy and 51 fs/52 fs peak-peak drift (in 7 days/17 h with the high power RF system, respectively) while the reference phase varies both ∼30 ps. An ∼4 h beam-based experiment has also been conducted to evaluate the overall performance of the whole XGLS timing and synchronization system, which shows that the PRDS, LLRF system, high power RF system, and laser oscillator laser-RF synchronization system can keep long-term phase stability.