We consider a model Hamiltonian fitted on the ab-initio band structure to describe the electron-phonon coupling between the electronic $\sigma-$bands and the phonon E$_{2g}$ mode in MgB$_2$. The model allows for analytical calculations and numerical treatments using very large k-point grids. We calculate the phonon self-energy of the E$_{2g}$ mode along two high symmetry directions in the Brillouin zone. We demonstrate that the contribution of the $\sigma$ bands to the Raman linewidth of the E$_{2g}$ mode via the electron-phonon coupling is zero. As a consequence the large resonance seen in Raman experiments cannot be interpreted as originated from the $E_{2g}$ mode at $\Gamma$. We examine in details the effects of Fermi surface singularities in the phonon spectrum and linewidth and we determine the magnitude of finite temperature effects in the the phonon self-energy. From our findings we suggest several possible effects which might be responsible for the MgB$_2$ Raman spectra.