We present a theory for the rate of energy exchange between electrons and ions-also known as the electron-ion coupling factor-in physical systems ranging from hot solid metals to plasmas, including liquid metals and warm dense matter. The paper provides the theoretical foundations of a recent work [J. Simoni and J. Daligault, Phys. Rev. Lett. 122, 205001 (2019)PRLTAO0031-900710.1103/PhysRevLett.122.205001], where first-principles quantum molecular dynamics calculations based on this theory were presented for representative materials and conditions. We first derive a general expression for the electron-ion coupling factor that includes self-consistently the quantum mechanical and statistical nature of electrons, the thermal and disorder effects, and the correlations between particles. The electron-ion coupling is related to the friction coefficients felt by individual ions due to their nonadiabatic interactions with the electrons. Each coefficient satisfies a Kubo relation given by the time integral of the autocorrelation function of the interaction force of an ion with the electrons. Exact properties and different representations of the general expressions are discussed. We then show that our theory reduces to well-known models in limiting cases. In particular, we show that it simplifies to the standard electron-phonon coupling formula in the limit of hot solids with lattice and electronic temperatures much greater than the Debye temperature, and that it extends the electron-phonon coupling formula beyond the harmonic phonon approximation. For plasmas, we show that the theory readily reduces to the well-known Spitzer formula in the hot plasma limit, to the Fermi "golden rule" formula in the limit of weak electron-ion interactions, and to other models proposed to go beyond the latter approximation. We explain that the electron-ion coupling is particularly well adapted to average atom models, which offer an effective way to include nonideal interaction effects to the standard models and at a much reduced computational cost in comparison to first-principles quantum molecular dynamics simulations.