We study the four-point function of the lowest-lying half-BPS operators in the mathcal{N} = 4 SU(N) super-Yang-Mills theory and its relation to the flat-space four-graviton amplitude in type IIB superstring theory. We work in a large-N expansion in which the complexified Yang-Mills coupling τ is fixed. In this expansion, non-perturbative instanton contributions are present, and the SL(2, ℤ) duality invariance of correlation functions is manifest. Our results are based on a detailed analysis of the sphere partition function of the mass-deformed SYM theory, which was previously computed using supersymmetric localization. This partition function determines a certain integrated correlator in the undeformed mathcal{N} = 4 SYM theory, which in turn constrains the four-point correlator at separated points. In a normalization where the two-point functions are proportional to N2− 1 and are independent of τ and overline{tau} , we find that the terms of order sqrt{N} and 1/sqrt{N} in the large N expansion of the four-point correlator are proportional to the non-holomorphic Eisenstein series Eleft(frac{3}{2},tau, overline{tau}right) and Eleft(frac{5}{2},tau, overline{tau}right) , respectively. In the flat space limit, these terms match the corresponding terms in the type IIB S-matrix arising from R4 and D4R4 contact inter-actions, which, for the R4 case, represents a check of AdS/CFT at finite string coupling. Furthermore, we present striking evidence that these results generalize so that, at order {N}^{frac{1}{2}-m} with integer m ≥ 0, the expansion of the integrated correlator we study is a linear sum of non-holomorphic Eisenstein series with half-integer index, which are manifestly SL(2, ℤ) invariant.