The late-time behavior of spectral form factor (SFF) encodes the inherent discreteness of a quantum system, which should be generically nonvanishing. We study an index analog of the microcanonical spectrum form factor in four-dimensional N=4 super Yang-Mills theory. In the large N limit and at large enough energy, the most dominant saddle corresponds to the black hole in the AdS bulk. This gives rise to the slope that decreases exponentially for a small imaginary chemical potential, which is a natural analog of an early time. We find that the "late-time" behavior is governed by the multicut saddles that arise in the index matrix model, which are nonperturbatively subdominant at early times. These saddles become dominant at late times, preventing the SFF from decaying. These multicut saddles correspond to the orbifolded Euclidean black holes in the AdS bulk, therefore giving the geometrical interpretation of the "ramp". Our analysis is done in the standard AdS/CFT setting without ensemble average or wormholes.