Anisotropy in crystal structure has provided a new degree of freedom to tune the physical and photoelectric properties of low-symmetrical materials, enabling the anisotropic band structure and polarization-resolved applications. Thus, the identification and control on the anisotropy is of great importance for the development of polarization-integrated devices. In this work, we introduce a promising two-dimensional (2D) dimetal chalcogenide Ta2NiSe5 with strong in-plane anisotropic structure and explore the effects of thickness, temperature and substrate on its in-plane anisotropy via an angle-resolved polarized Raman spectra. It is shown that the Raman mode softens with increasing temperature, revealing the anharmonic phonon properties of the Ta2NiSe5 flakes. We found that the anisotropy of phonon vibration is strongly related with the thickness, temperature and used substrate, which is more obvious at thinner samples, higher temperature and opaque silicon substrate compared to sapphire and suspended conditions. This work first reveals the influence factor on the anisotropy of phonon mode, which will guide the fundamental research of anisotropic physics and experimental design for the angle-resolved electronics.