ScVO4 and ScPO4 represent the zircon (xenotime) type structures with smallest trivalent cations, and that enable them to host both transition metal and rare-earth ions for applied optical materials. Thus, their crystal chemistry and thermophysical properties becomes relevance for their application in non-ambient conditions. In this report, high temperature crystal chemistry and vibrational properties of ScVO4 and ScPO4, as observed from in situ high temperature powder XRD and Raman spectroscopic studies, are reported. The comparative analyses of the results indicate that, though both are isostructural, they show drastically different thermal expansion behavior. In case of ScVO4, the c-axis shows significantly larger expansion compared to a-axis, while in ScPO4 the thermal expansion along and a and c-axes are more or less similar. At ambient condition, the thermal expansion anisotropy in ScPO4 and ScVO4 are 1.02 and 3.97, respectively. Additionally, ScPO4 shows relatively lower coefficient of volume thermal expansion compared to ScVO4, (αv = 23.64 × 10−6 K−1 for ScPO4 and 26.09 × 10−6 K−1 for ScVO4), and is contributed by the expansion of ScO8 units in their structures. The thermal expansion coefficients of ScO8 unit in ScPO4 and ScVO4 are 36.3 × 10−6 K−1 and 39.1 × 10−6 K−1, respectively. Temperature evolution of Raman modes indicates weakening of all the modes, except a symmetric stretching mode, with increasing temperature. The anharmonic analyses of the Raman modes indicate that implicit contributions in ScVO4 and ScPO4 are appreciably higher than the explicit contributions, and hence the changes in mode wavenumbers with volume play dominating role in governing their thermal expansion behaviors. Further, it is concluded that ScPO4 is characterized by more or less like rigid unit cell compared to ScVO4.