This paper presents a novel time-to-amplitude converter (TAC) for single-photon direct time-of-flight (d-TOF) measurement. An innovative high impedance switch approach is proposed to significantly improve the stability of the integral current and to increase the swing of the timing voltage, enabling high linearity together with low power consumption on TAC. Implemented in a standard $0.18~\mu \text{m}$ CMOS technology, the TAC occupies only an area of $100~\mu \text{m}^{2}$ , featuring a compact configuration. The experimental results indicate that the presented TAC can acquire a 78 ps timing resolution in a 20 ns full-scale range (FSR) under the supply of 1.8 V. In particular, the TAC gains an excellent linearity characteristic with the very low differential nonlinearity (DNL) and integral nonlinearity (INL) within ±0.1 LSB and ±0.15 LSB, respectively. Additionally, the extremely low dynamic power consumption of $20~\mu \text{W}$ and the high pixel fill factor of 16 % are also achieved simultaneously. Thanks to the outstanding advantages of high-performance and compactness, the proposed TAC may be a promising candidate to realize high-density compact TOF-based imagers.