A scheme is presented for realizing a two-qubit quantum SWAP gate for intracavity fields. In the scheme, a $\mathit{V}$-type atomic ensemble prepared in its ground state mediates the interaction between the two-cavity modes. Under certain conditions, the cavity-field degree of freedom is decoupled from the atomic degrees of freedom, and the effective coupling strength between the two-cavity modes scales up with $\sqrt{N}$ ($\mathit{N}$ is the number of effective atoms). The numerical simulation shows that the quantum SWAP gate may have a high fidelity under the influence of the atomic spontaneous emission and the decay of the cavity modes.