A detailed study of two-photon exchange in unpolarized and polarized elastic electron-nucleon scattering is presented, taking particular account of nucleon finite size effects. Contributions from nucleon elastic intermediate states are found to have a strong angular dependence, which leads to a partial resolution of the discrepancy between the Rosenbluth and polarization transfer measurements of the proton electric to magnetic form factor ratio, ${G}_{E}/{G}_{M}$. The two-photon exchange contribution to the longitudinal polarization transfer ${P}_{L}$ is small, whereas the contribution to the transverse polarization transfer ${P}_{T}$ is enhanced at backward angles by several percent, increasing with ${Q}^{2}$. This gives rise to a small, $\ensuremath{\lesssim}3%$ suppression of ${G}_{E}/{G}_{M}$ obtained from the polarization transfer ratio ${P}_{T}/{P}_{L}$ at large ${Q}^{2}$. We also compare the two-photon exchange effects with data on the ratio of ${e}^{+}p$ to ${e}^{\ensuremath{-}}p$ cross sections, which is predicted to be enhanced at backward angles. Finally, we evaluate the corrections to the form factors of the neutron and estimate the elastic intermediate state contribution to the $^{3}\mathrm{He}$ form factors.