Semiconductor quantum dots have emerged as an especially promising platform for the generation of polarization-entangled photon pairs. However, it was demonstrated recently that the two-photon excitation scheme employed in state-of-the-art experiments limits the achievable degree of entanglement by introducing which-path information. In this work, the combined impact of two-photon excitation and longitudinal acoustic phonons on photon pairs emitted by strongly-confining quantum dots is investigated. It is found that phonons further reduce the achievable degree of entanglement even in the limit of vanishing temperature due to phonon-induced pure dephasing and phonon-assisted one-photon processes, which increase the reexcitation probability. In addition, the degree of entanglement, as measured by the concurrence, decreases with rising temperature and/or pulse duration, even if the excitonic fine-structure splitting is absent and when higher electronic states are out of reach. Furthermore, in the case of finite fine-structure splittings, phonons enlarge the discrepancy in concurrence for different laser polarizations.