We establish systematic consolidation of the Aharonov-Bohm and Aharonov-Casher effects including their scalar counterparts. Their formal correspondences in acquiring topological phases are revealed on the basis of the gauge symmetry in non-simply connected spaces and the adiabatic condition for the state of magnetic dipoles. In addition, investigation of basic two-body interactions between an electric charge and a magnetic dipole clarifies their appropriate relative motions and discloses physical interrelations between the effects. Based on the two-body interaction, we also construct an exact microscopic description of the Aharonov-Bohm effect, where all the elements are treated on equal footing, i.e., magnetic dipoles are described quantum-mechanically and electromagnetic fields are quantized. This microscopic analysis not only confirms the conventional (semiclassical) results and the topological nature but also allows one to explore the fluctuation effects due to the precession of the magnetic dipoles with the adiabatic condition relaxed.