The interaction of an atomic system with a single mode of a quantized electromagnetic field is investigated in the dipole approximation. Two approximations that are almost always made in this analysis are the neglect of the diamagnetic term in the Hamiltonian and the dropping of the rotating-wave approximation (RWA) terms that correspond to processes that do not conserve energy. Both approximations involve the neglect of terms that are of the same order of magnitude so that consistency demands that they be considered together. It is demonstrated that, in the dipole approximation, a minor modification of the usual application of quantum electrodynamics to any atomic system can exactly incorporate the effects of the diamagnetic term into a new frequency of the electromagnetic cavity mode; this frequency is slightly different from that of the empty cavity. This result is then applied to the specific case of a charged harmonic oscillator interacting with a single mode. For this simple model an analytic solution can then be found that includes the effects of the diamagnetic term and does not use the RWA. It is found that, for off resonance, the normal modes of the coupled system oscillate at shifted frequencies which are in agreement with the second-order perturbation-theory calculation of the Lamb shift. When the cavity mode is tuned to the atomic oscillator frequency, the perturbation-theory Lamb-shift calculation is no longer valid, and the exact solution derived in this paper predicts a resonant Lamb shift that is significantly larger than the off-resonant single-mode Lamb shift. Furthermore, the frequency spectrum is qualitatively changed from the single frequency of the free atomic system to a doublet spectrum. An analytic solution of this harmonic-oscillator problem can also be found when the RWA terms are dropped from the Hamiltonian. Comparison of the two solutions reveals that the RWA terms contribute a frequency shift of magnitude independent of the field intensity in the simple case of a charged harmonic oscillator.When trajectories of the exact and RWA solutions are compared in phase space, it is found that one effect of the RWA terms is to lift a twofold degeneracy of the phase-space orbits that occurs when the RWA terms are neglected. The harmonic-oscillator problem has several features in common with the interaction of a two-level atom with a single quantized field mode. Both Hamiltonians exhibit inversion symmetry and hence have parity as a good quantum number. Both Hamiltonians contain RWA terms. When these RWA terms are neglected, both Hamiltonians have a new constant of motion that can be identified with the total number of excitations in the combined system.