Silicon photonics (SiPh) technology has facilitated the deployment of integrated photonics across different application domains, from ultra-fast communication in Datacom applications to energy-efficient optical computation in emerging hardware accelerators for machine learning. More recently, the integration of SiPh and phase change materials has created a unique opportunity to realize adaptable, reconfigurable, and programmable photonic platforms. In particular, the nonvolatile programmability in phase change materials has made them a promising candidate for implementing photonic memory cells and architectures. Accordingly, photonic memory systems and even in-memory photonic computing paradigms are on the rise, especially given their potential for improving data access in electronic and photonic processors. However, there are still many challenges in the design and fabrication of phase-change photonic integrated circuits, which need to be addressed. This article presents a comprehensive survey on the recent advances and challenges for the integration of phase change materials with contemporary photonic devices while focusing on the photonic memory application. In particular, we explore phase-change photonic memory from the material level to the architecture level by presenting an overview of different material-level characteristics of phase change materials with their optical, electrical, and thermal properties as well as their integration into SiPh devices and photonic memory architectures and their application for in-memory photonic computing. We also present a comparison with electronic memory and discuss open research challenges that must be addressed to further advance phase-change photonic memory towards successful integration into emerging computing systems.