We develop a new approach to extracting the physical consequences of S-duality of four-dimensional mathcal{N} = 4 super Yang-Mills (SYM) and its string theory dual, based on SL(2, ℤ) spectral theory.We observe that CFT observables \U0001d4aa, invariant under SL(2, ℤ) transformations of a complexified gauge coupling τ, admit a unique spectral decomposition into a basis of square-integrable functions. This formulation has direct implications for the analytic structure of mathcal{N} = 4 SYM data, both perturbatively and non-perturbatively in all parameters. These are especially constraining for the structure of instantons: k-instanton sectors are uniquely determined by the zero- and one-instanton sectors, and Borel summable series around k-instantons have convergence radii with simple k-dependence. In large N limits, we derive the existence and scaling of non-perturbative effects, in both N and the ‘t Hooft coupling, which we exhibit for certain mathcal{N} = 4 SYM observables. An elegant benchmark for these techniques is the integrated stress tensor multiplet four-point function, conjecturally determined by [1] for all τ for SU(N) gauge group; we elucidate its form, and explain how the SU(2) case is the simplest possible observable consistent with SL(2, ℤ)-invariant perturbation theory.These results have ramifications for holography. We explain how leftlangle mathcal{O}rightrangle , the ensemble average of \U0001d4aa over the mathcal{N} = 4 supersymmetric conformal manifold with respect to the Zamolodchikov measure, is cleanly isolated by the spectral decomposition. We prove that the large N limit of leftlangle mathcal{O}rightrangle equals the large N, large ‘t Hooft coupling limit of \U0001d4aa. Holographically speaking, leftlangle mathcal{O}rightrangle = \U0001d4aasugra, its value in type IIB supergravity on AdS5× S5. This result, which extends to all orders in 1/N, embeds ensemble averaging into the traditional AdS/CFT paradigm. The statistics of the SL(2, ℤ) ensemble exhibit both perturbative and non-perturbative 1/N effects. We discuss further implications and generalizations to other AdS compactifications of string/M-theory.