The resistive wall impedance of a vacuum chamber with elliptic cross section is of particular interest for circular particle accelerators as well as for undulators in free electron lasers. By using the electric field of a point charge and of a small dipole moving at arbitrary speed in an elliptical vacuum chamber, expressed in terms of Mathieu functions, in this paper we take into account the finite conductivity of the beam pipe walls by means of the surface impedance, and evaluate the longitudinal and transverse driving and detuning impedances for any beam velocity. We also extend the definition of the Yokoya form factors, valid in the thick wall regime, at any beam energy, and show that, in the ultra-relativistic limit, they coincide with the ones that are found in literature. The method is also extended to the multilayer vacuum chamber case. Under conditions generally satisfied with particle accelerator beam pipes, the classical transmission line theory can be used to modelling the impedance seen by a bunch in a vacuum chamber with several layers as an equivalent circuit with the same number of load impedances, giving, as result, a surface impedance that can be used in combination with the fields of the elliptic geometry to obtain the resistive wall impedance in an elliptical multilayer vacuum chamber. The results are also compared with a more time consuming 3D electromagnetic code and with solutions for known cases of circular and flat beam pipe.