Let f be a Morse function on a smooth compact manifold M with boundary. The path component mathrm {PH}^{-1}_f(D) containing f of the space of Morse functions giving rise to the same Persistent Homology D=mathrm {PH}(f) is shown to be the same as the orbit of f under pre-composition phi mapsto fcirc phi by diffeomorphisms of M which are isotopic to the identity. Consequently we derive topological properties of the fiber mathrm {PH}^{-1}_f(D): In particular we compute its homotopy type for many compact surfaces M. In the 1-dimensional settings where M is the unit interval or the circle we extend the analysis to continuous functions and show that the fibers are made of contractible and circular components respectively.