Bang-bang phase-locked loops (BBPLLs) are inherently nonlinear due to the hard nonlinearity introduced by the binary phase detector (BPD). This paper provides an exact statistical analysis of the steady-state timing jitter in a first-order BBPLL when the reference clock is subject to accumulative jitter. By elaborating on the analogy of viewing a first-order BBPLL as a single-integration delta modulator (DM) in the phase domain, we are able to relate hunting jitter and slew-rate limiting in a BBPLL to granular noise and slope overload in a DM. The stochastic timing-jitter behavior is modeled as a sign-dependent random walk, for which we obtain the asymptotic characteristic function and analytical expressions for the first four cumulants. These expressions are applied to the BBPLL to statistically analyze the static timing offset and the rms timing jitter, including the effect of a frequency offset. The analysis shows that the rms timing jitter is constant for small rms clock jitter and grows quadratically with large rms clock jitter, and that there exists an optimal bang-bang phase step for minimum rms timing jitter. Computing the kurtosis reveals the effect of the BPD nonlinearity: The timing jitter is largely non-Gaussian.