Head-related transfer functions (HRTFs) describe the filtering of the incoming sound by the human anatomy. They contain the so-called broadband time-of-arrivals (TOAs), which interaural differences yield the well-known interaural time differences used to estimate the lateral position of sound sources by the human auditory system. The TOAs are essential for a time-synchronous binaural rendering of multiple virtual sound sources or for interpolation of the timing information in an existing HRTF set. Estimation of the TOA is usually done separately for each spatial direction and is thus prone to errors and directional outliers. A method for a robust estimation of spatially continuous TOA function from a set of listener-specific HRTFs is presented. The method relies on a geometric model of the HRTF-measurement setup represented by parameters like head position, radius, and ear position. The model parameters were fit to HRTFs of a sphere numerically calculated under various conditions, and to measured HRTFs of 160 listeners. The resulting model parameters and TOA functions corresponded well with the measurement geometry and manually derived TOAs, respectively. The model parameters were further compared to those resulting from a simplified model which assumes the listener being in the center of the HRTF-measurement setup, demonstrating the impact of the usually neglected aspect of listener position on the HRTF timing quality.