To any unramified double cover $\pi:\tilde C \to C$ of projective irreducible and nonsingular curves one associates the Prym variety $P = P(\pi)$. For $C$ nonhyperelliptic of genus $g \geq 6$ we consider the natural embedding $\tilde C \subset P$ (defined up to translation) of $\tilde C$ into $P$ and we study the local structure of the Hilbert scheme $Hilb^P$ of $P$ at the point $[\tilde C]$. We show that this structure is related with the local geometry of the Prym map, or more precisely with the validity of the infinitesimal version of Torelli's theorem for Pryms at $[\pi]$