Yeast colonies are routinely grown on agar plates in everyday experimental settings to understand basic molecular processes, produce novel drugs, improve health, and so on. Standardized conditions ensure these colonies grow in a reproducible fashion, while in nature microbes are under a constantly changing environment. Here we combine the power of computational simulations and laboratory experiments to investigate the impact of non-standard environmental factors on colony growth. We present the developement and parameterization of a quantitative agent-based model for yeast colony growth to reproduce measurements on colony size and cell number in a colony at non-standard environmental conditions. Specifically, we establish experimental conditions that mimic the effects of humidity changes and nutrient gradients. Our results show how colony growth is affected by moisture changes, nutrient availability, and initial colony inoculation conditions. We show that initial colony spread, not initial cell number have higher impact on the final size and cell number of colonies. Parameters of the model were identified by fitting these experiments and the fitted model gives guidance to establish conditions which enable unlimited growth of yeast colonies.