Information on factors limiting the shelf life of extended shelf life (ESL) milk produced by microfiltration and subsequent pasteurization is very limited. In this study, three different batches of ESL milk were analyzed at different stages of the production process and during storage at 4 °C, 8 °C and 10 °C in order to evaluate the changes in bacterial cell counts, microbial diversity and enzymatic quality. Additionally, detailed biodiversity analyses of 250 retail ESL milk packages produced by five manufacturers in Germany, Austria and Switzerland were performed at the end of shelf life. It was observed that microfiltration decreased the microbial loads by 5-6 log₁₀ units to lower than 1 CFU/mL. However, bacterial counts at the end of shelf life were extremely variable and ranged between <1 and 8 log₁₀ CFU/mL. 8% of all samples showed spoilage indicated by cell counts higher than 6 log₁₀ CFU/mL. The main spoilage groups of bacteria were Gram-negative post-process recontaminants (Acinetobacter, Chryseobacterium, Psychrobacter, Sphingomonas) and the spore formers Paenibacillus and Bacillus cereus, while other spore formers and Microbacterium spp. did not reach spoilage levels. Paenibacillus spp. and B. cereus apparently influenced enzymatic spoilage, as indicated by increased free fatty acid production, pH 4.6 soluble peptide fractions and off-flavors. In some cases, enzymatic spoilage was observed although microbial counts were well below 6 log₁₀ CFU/mL. Thirteen B. cereus isolates were characterized for their toxin profiles and psychrotolerance. Hbl, nhe, and cytK toxin genes were detected in ten, thirteen, and four isolates, respectively, whereas the ces gene was always absent. Interestingly, only three of the thirteen isolates could be allocated to psychrotolerant genotypes, as indicated by the major cold shock cspA gene signature. Generally, large discrepancies in microbial loads and biodiversity were observed at the end of shelf life, even among packages of the same production batch. We suggest that such unexpected differences may be due to very low cell counts after ESL treatment, causing stochastic variations of initial species distributions in individual packages. This would result in the development of significantly different bacterial populations during cold storage, including the occasional development of high numbers of pathogenic species such as B. cereus or Acinetobacter.