We present new ISOCAM mid-infrared spectra of three starbursting nearby dwarf galaxies, NGC1569, IIZw40, NGC1140 and the 30Dor region of the LMC and explore the properties of the ISM in low-metallicity environments, also using additional sources from the literature. We analyse the various components of the ISM probed by the mid-infrared observations and compare them with other Galactic and extragalactic objects. The MIR spectra of the low-metallicity starburst sources are dominated by the [NeIII] and [SIV] lines, as well as a steeply rising dust continuum. PAH bands are generaly faint, both locally and averaged over the full galaxy, in stark contrast to dustier starburst galaxies, where the PAH features are very prominant and even dominate on global scales. The hardness of the modeled interstellar radiation fields for the dwarf galaxies increases as the presence of PAH band emission becomes less pronounced. The [NeIII]/[NeII] ratios averaged over the full galaxy are strikingly high, often >10. Thus, the hard radiation fields are pronounced and pervasive. We find a prominent correlation between the PAHs/VSGs and the [NeIII]/[NeII] ratios for a wide range of objects, including the low metallicity galaxies as well as Galactic HII regions and other metal-rich galaxies. This effect is consistent with the hardness of the interstellar radiation field playing a major role in the destruction of PAHs in the low metallicity ISM. We see a PAHs/VSGs and metallicity correlation, also found by Engelbracht et al. (2005) for a larger survey. Combined effects of metallicity and radiation field seem to be playing important roles in the observed behavior of PAHs in the low metallicity systems.