We derive the fivebrane instanton corrections to the hypermultiplet moduli space $M_H$ of Calabi-Yau string vacua using S-duality symmetry of the type IIB formulation. The result is given in terms of a set of holomorphic functions on the twistor space of $M_H$. It contains not only all orders of the instanton expansion, but also takes into account the presence of D1-D(-1)-brane instantons. Furthermore, we provide a thorough study of the group of discrete isometries of $M_H$ and show that its closure requires a modification of certain symmetry transformations. After this modification, the fivebrane instantons are proven to be consistent with the full duality group.