Bulk metallic glass (BMG), metastable quasicrystals (QCs) and two types of approximant phases concurrently form from the melt during cooling at different rates in a Mg–Zn–Yb model alloy. Such an enriched phase competition between various metastable and stable crystals has also been found during the heating of Mg–Zn–Yb metallic glass between the glass transition and liquidus temperatures. Metastable QC acts as a transient phase that nucleates first from the melt and subsequently transforms into an equilibrium approximant phase. During heating, a metastable approximant phase transforms into the Mg-QC phase mixture which then transforms into the Mg and equilibrium approximant eutectic microstructure. Nucleation kinetics of QCs is critical to analyze for further understanding the relation between QC formation and glass forming ability of the alloy. For this purpose, a metastable phase diagram for QCs has been determined using a novel experimental strategy via fast differential scanning calorimetry (FDSC). Along compositional line binding Mg to Zn17Yb3, one stable eutectic point, i.e. Mg68.6Zn26.7Yb4.7 at 390°C, and one metastable eutectic composition, i.e. Mg64.6Zn30.1Yb5.3 at 350°C, were determined during interrupted heating experiments. We have demonstrated that the glass forming ability of Mg–Zn–Yb alloys strongly depends on the metastable phase diagram of QCs and their nucleation kinetics. Nucleation of metastable QCs limits the metallic glass formation, therefore the bulk metallic glass forming compositional region was found to be below metastable hypoeutectic compositions.