A realistic interatomic potential of Mg–Ca–Ag system is constructed under a proposed modified tight-binding scheme with the aid of ab initio calculations. Then, the favored and optimized compositions for forming Mg–Ca–Ag metallic glasses are investigated by means of the potential-based molecular dynamics simulations of atomistic modeling. The calculation shows that the Mg–Ca–Ag ternary system is liable to form amorphous alloys and the favored compositions for metallic glass formation locate at a sub-region, where the amorphization driving force is greater than that of other regions. Notably, we improve the conventional method of removing the facets in the amorphous structure analysis, and we introduce K-means clustering to eliminate the facets of the Voronoi polyhedra, which allows more reasonable description of various geometries of the different central atoms and their neighboring atoms. Then, both the Voronoi tessellation method and the Honeycutt–Andersen pair analysis are used to describe the disordered structure in Mg50Ca30Ag20 metallic glass based on the result of K-means clustering. Our work provides good guidance for composing the Mg–Ca–Ag metallic glasses, and further verifies the microstructure of the amorphous alloys.