Improvements in temporal resolution of single-photon detectors enable increased data rates and transmission distances for both classical and quantum optical communication systems, higher spatial resolution in laser ranging, and observation of shorter-lived fluorophores in biomedical imaging. In recent years, superconducting nanowire single-photon detectors (SNSPDs) have emerged as the most efficient time-resolving single-photon-counting detectors available in the near-infrared, but understanding of the fundamental limits of timing resolution in these devices has been limited due to a lack of investigations into the timescales involved in the detection process. We introduce an experimental technique to probe the detection latency in SNSPDs and show that the key to achieving low timing jitter is the use of materials with low latency. By using a specialized niobium nitride SNSPD we demonstrate that the system temporal resolution can be as good as 2.6 ± 0.2 ps for visible wavelengths and 4.3 ± 0.2 ps at 1,550 nm. Knowledge about detection latency provides a guideline to reduce the timing jitter of niobium nitride superconducting nanowire single-photon detectors. A timing jitter of 2.6 ps at visible wavelength and 4.3 ps at 1,550 nm is achieved.