The ability to detect fast responses with functional MRI depends on the speed of hemodynamic responses to neural activity, because hemodynamic responses act as a temporal low-pass filter which blurs rapid changes. However, the shape and timing of hemodynamic responses are highly variable across the brain and across stimuli. This heterogeneity of responses implies that the temporal specificity of fMRI signals, or the ability of fMRI to preserve fast information, could also vary substantially across the cortex. In this work we investigated how local differences in hemodynamic response timing affect the temporal specificity of fMRI. We used ultra-high field (7T) fMRI at high spatiotemporal resolution, studying the primary visual cortex (V1) as a model area for investigation. We used visual stimuli oscillating at slow and fast frequencies to probe the temporal specificity of individual voxels. As expected, we identified substantial variability in temporal specificity, with some voxels preserving their responses to fast neural activity more effectively than others. We investigated which voxels had the highest temporal specificity, and tested whether voxel timing was related to anatomical and vascular features. We found that low temporal specificity is only weakly explained by the presence of large veins or cerebral cortical depth. Notably, however, temporal specificity depended strongly on a voxel's position along the anterior-posterior anatomical axis of V1, with voxels within the calcarine sulcus being capable of preserving close to 25% of their amplitude as the frequency of stimulation increased from 0.05Hz to 0.20Hz, and voxels nearest to the occipital pole preserving less than 18%. These results indicate that detection biases in high-resolution fMRI will depend on the anatomical and vascular features of the area being imaged, and that these biases will differ depending on the timing of the underlying neuronal activity. While we attribute this variance primarily to hemodynamic effects, neuronal nonlinearities may also influence response timing. Importantly, this spatial heterogeneity of temporal specificity suggests that it could be exploited to achieve higher specificity in some locations, and that tailored data analysis strategies may help improve the detection and interpretation of fast fMRI responses.