We investigate the zero-bias behavior of Josephson junctions made of encapsulated graphene boron nitride heterostructures in the long ballistic junction regime. For temperatures down to $2.7$ K, the junctions appear nonhysteretic with respect to the switching and retrapping currents ${I}_{C}$ and ${I}_{R}$. A small nonzero resistance is observed even around zero-bias current and scales with temperature as dictated by the phase diffusion mechanism. By varying the graphene carrier concentration we are able to confirm that the observed phase diffusion mechanism follows the trend for an overdamped Josephson junction. This is in contrast with the majority of graphene-based junctions which are underdamped and shorted by the environment at high frequencies.