The grain boundary diffusion process (GBDP) has been widely applied to increase the coercivity of Nd-Fe-B magnets. After GBDP with Dy/Tb-rich diffusion sources the thickness of the Dy/Tb-rich shell formed on the epitaxial layer of the 2:14:1 main phase grain decreases from the magnet surface to the center. However, the influence of the Dy/Tb-rich shell gradient distribution on magnetic properties has not been thoroughly studied. In this work, a sintered Nd-Fe-B magnet was subjected to GBDP with Pr60Tb10Cu30 alloy at 860°C for various diffusion times (3 h, 6 h and 9 h). The coercivity improves rapidly from 884 kA/m (without GBDP) to 1533 kA/m after GBDP of 3 h. The coercivity further increases to 1741 kA/m with increased diffusion time to 6 h. But only marginal coercivity enhancement (rising to 1803 kA/m) can be obtained by further prolonging the diffusion time to 9 h. Microstructure analysis indicates that the long diffusion time leads to the surface grain coarsening, which degrades the diffusion efficiency. Meanwhile, micromagnetic simulation indicates that if the thickness of the Tb-rich shell in magnet center is less than 4 nm, the coercivity increases significantly with the enhanced thickness uniformity of the Tb-rich shell. But if the thickness of the Tb-rich shell in magnet center is higher than 4 nm, the coercivity cannot be improved effectively by further increasing the thickness uniformity of the Tb-rich shell. The results in this work clarify the mechanism of the magnetic property dependence on the diffusion time and help to optimize the GBDP parameters in the future.