In this paper, we consider some second-order effective Hamiltonians describing the interaction of the quantum electromagnetic field with atoms or molecules in the nonrelativistic limit. Our procedure is valid only for off-energy-shell processes, specifically virtual processes such as those relevant for ground-state energy shifts and dispersion van der Waals and Casimir-Polder interactions, while on-energy-shell processes are excluded. These effective Hamiltonians allow for a considerable simplification of the calculation of radiative energy shifts, dispersion, and Casimir-Polder interactions, including in the presence of boundary conditions. They can also provide clear physical insights into the processes involved. We clarify that the form of the effective Hamiltonian depends on the field states considered, and consequently different expressions can be obtained, each of them with a well-defined range of validity and possible applications. We also apply our results to some specific cases, mainly the Lamb shift, the Casimir-Polder atom-surface interaction, and the dispersion interactions between atoms, molecules, or, in general, polarizable bodies.