SN 2002cx-like Type Ia supernovae (also known as SNe Iax) represent one of the most numerous peculiar SN classes. They differ from normal SNe Ia by having fainter peak magnitudes, faster decline rates and lower photospheric velocities, displaying a wide diversity in these properties. We present both integral-field and long-slit visual-wavelength spectroscopy of the host galaxies and explosion sites of SNe Iax to provide constraints on their progenitor formation scenarios. The SN Iax explosion site metallicity distribution is similar to that of core-collapse (CC) SNe and metal-poor compared to normal SNe Ia. Fainter members, speculated to form distinctly from brighter SN Iax, are found at a range of metallicities, extending to very metal-poor environments. Although the SN Iax explosion sites' ages and star-formation rates are comparatively older and less intense than the distribution of star forming regions across their host galaxies, we confirm the presence of young stellar populations (SP) at explosion environments for most SNe Iax, expanded here to a larger sample. Ages of the young SP (several $\times 10^{7}$ to $10^8$~yrs) are consistent with predictions for young thermonuclear and electron-capture SN progenitors. The lack of extremely young SP at the explosion sites disfavours very massive progenitors such as Wolf-Rayet explosions with significant fall-back. We find weak ionised gas in the only SN Iax host without obvious signs of star-formation. The source of the ionisation remains ambiguous but appears unlikely to be mainly due to young, massive stars.