We use a recently-developed analytic model for the ISM structure from scales of GMCs through star-forming cores to explore how the pre-stellar core mass function (CMF) and, by extrapolation, stellar initial mass function (IMF) should depend on both local and galactic properties. If the ISM is supersonically turbulent, the statistical properties of the density field follow from the turbulent velocity spectrum, and the excursion set formalism can be applied to analytically calculate the mass function of collapsing cores on the smallest scales on which they are self-gravitating (non-fragmenting). Two parameters determine the model: the disk-scale Mach number M_h (which sets the shape of the CMF), and the absolute velocity (to assign an absolute scale). For 'normal' variation in disk properties and core gas temperatures in the MW and local galaxies, there is almost no variation in the predicted high-mass behavior of the CMF/IMF. The slope is always close to Salpeter down to <1 M_sun. We predict modest variation in the sub-solar regime, mostly from variation in M_h, but within the observed scatter in sub-solar IMFs in local regions. For fixed galaxy properties, there is little variation in shape or 'upper mass limit' with parent GMC mass. However, in extreme starbursts (e.g. ULIRGs) we predict a bottom-heavy CMF. This agrees with the IMF inferred for the centers of Virgo ellipticals, believed to form in such a nuclear starburst. The CMF is bottom heavy despite the gas temperature being an order of magnitude larger, because M_h is also much larger. Larger M_h values make the 'parent' cloud mass (turbulent Jeans mass) larger, but promote fragmentation to smaller scales; this steepens the slope of the low-mass CMF and shifts the turnover mass. The model may predict a top-heavy CMF for the sub-pc disks around Sgr A*, but the relevant input parameters are uncertain.