Surface integral equation (SIE) methods are of great interest for the efficient electromagnetic modeling of various devices, from integrated circuits to antenna arrays. Existing acceleration algorithms for SIEs, such as the adaptive integral method (AIM), enable the fast approximation of interactions between well-separated mesh elements. Nearby interactions involve the singularity of the kernel, and must instead be computed accurately with direct integration at each frequency of interest, which can be computationally expensive. We propose a novel algorithm for reducing the cost-per-frequency of near-region computations for both homogeneous and layered background media. In the proposed extended AIM (AIMx), the SIE operators are decomposed into a frequency-independent term containing the singularity of the kernel, and a nonsingular frequency-dependent term. Direct integration is only required for the frequency-independent term, and can be reused at each frequency, leading to significantly faster frequency sweeps. The frequency-dependent term is captured with good accuracy via fast Fourier transform (FFT)-based acceleration even in the near region, as confirmed with an error analysis. The accuracy and efficiency of the proposed method are demonstrated through numerical examples drawn from several applications, and CPU times are significantly reduced by factors ranging from three to 16.