An efficient integral-equation method is presented for the fast analysis of scattering from arbitrarily shaped 3-D structures in a rectangular cavity. The proposed method employs: 1) the frequency-domain surface-volume electric field integral equation to model scattering from conductors and dielectrics; 2) the rectangular-cavity Green functions to account for the cavity walls; 3) the Ewald method and 3-D spatial interpolation to accelerate the Green function computations; and 4) the adaptive integral method (AIM) to reduce the computational complexity of the iterative method-of-moments solution procedure. The structure of interest is first meshed with arbitrary triangular/tetrahedral elements. The mesh is then enclosed with an auxiliary regular grid. Next, a four-step algorithm is executed: interpolation (mesh-to-grid), propagation (grid-to-grid), interpolation (grid-to-mesh), and correction (mesh-to-mesh). The computationally dominant propagation step of the AIM is accelerated by decomposing the Green functions into eight components that are in convolution or correlation form in the three Cartesian directions. This results in eight types of propagation matrices; these are in nested Toeplitz or Hankel form and are efficiently multiplied with the necessary vectors using 3-D fast Fourier transforms. Numerical results validate the method, quantify its costs, and demonstrate its utility.