We discuss the physics of moduli (light scalar fields with Planck-suppressed couplings to matter) in the case of low-scale supersymmetry breaking such as gauge mediation. We argue that even if the mechanism of moduli stabilization is decoupled from the mechanism of SUSY breaking, moduli masses will generically be parametrically related to the gravitino mass once the cancellation of the cosmological constant is taken into account. For low- scale SUSY breaking, this implies that moduli fields are light, long-lived relics that will generically drive the universe into a matter-dominated phase, in contradiction to standard BBN. We discuss two scenarios for evading this problem. The first is to consider very tuned supergravity potentials that can make the moduli heavy enough to decay at early times and reheat above the temperature of BBN. Viable cosmology can be achieved in this scenario, which has a population of highly relativistic light gravitinos arising from decays of moduli. Next, we consider the more natural scenario with light moduli. The saxion field associated with the solution of the strong CP problem provides the most natural candidate for driving thermal inflation, which dilutes the moduli. We construct an explicit model for this scenario, when the PQ symmetry breaking scale is larger than the messenger scale. The combination of the constraints on relic abundance and gamma-ray flux from decay of the moduli favors a particular region of SUSY breaking scale, 2 x 10^3 - 10^4 TeV. For either scenario, we find that it is generic for low-scale SUSY to be associated with late entropy production and accompanying low reheating temperatures.