Single-photon sources are a fundamental resource in quantum optics and quantum information science. Photons with differing spectral and temporal shapes do not interfere well and inhibit the performance of quantum applications such as linear optics quantum computing, boson sampling, and quantum networks. Indistinguishability and purity of photons emitted from different sources are crucial properties for many quantum applications. The ability to determine the state of single-photon sources therefore provides a means to assess their quality, compare different sources, and provide feedback for source tuning. Here, we propose and demonstrate a single-configuration experimental method enabling complete characterization of the spectral-temporal state of a pulsed single-photon source having both pure and mixed states. The method involves interference of the unknown single-photon source with a reference at a balanced beam splitter followed by frequency-resolved coincidence detection at the outputs. Fourier analysis of the joint-spectral two-photon interference pattern reveals the density matrix of the single-photon source in the frequency basis. We present an experimental realization of this method for pure and mixed state pulsed single-photon sources.