The stability of self-supporting dielectric film beam splitters at cryogenic temperature was studied with a test interferometer that allowed continuous visual monitoring of the surface figure. Test beam splitters using Mylar film remained flat to within a wavelength of visible light in a 3-cm diam central area during multiple cycles between room temperature and ~4 K. Factors that could affect the surface figure, such as assembly procedures and the mechanical design of the film holder, were studied to determine some of the conditions for employing Mylar beam splitters in cryogenic space instrumentation.