The quantification of the irregular morphology and distribution pattern of mineral grains is an essential but challenging task in ore-related mineralogical research, allowing for tracing the footprints of pattern-forming geological processes that are crucial to understanding mineralization and/or diagenetic systems. In this study, a large model, namely, the Segmenting Anything Model (SAM), was employed to automatically segment and annotate quartz, lepidolite and albite grains derived from Yichun rare-metal granite (YCRMG), based on which a series of fractal and multifractal methods, including box-counting calculation, perimeter–area analysis and multifractal spectra, were implemented. The results indicate that the mineral grains from YCRMG show great scaling invariance within the range of 1.04~52,300 μm. The automatic annotation of mineral grains from photomicrographs yields accurate fractal dimensions with an error of only 0.6% and thus can be utilized for efficient fractal-based grain quantification. The resultant fractal dimensions display a distinct distribution pattern in the diagram of box-counting fractal dimension (Db) versus perimeter–area fractal dimension (DPA), in which lepidolites are sandwiched between greater-valued quartz and lower-valued albites. Snowball-textured albites, i.e., concentrically arranged albite laths in quartz and K-feldspar, exhibit characteristic Db values ranging from 1.6 to 1.7, which coincide with the fractal indices derived from the fractal growth model. The zonal albites exhibit a strictly increasing trend regarding the values of fractal and multifractal exponents from core to rim, forming a featured “fractal-index banding” in the radar diagram. This pattern suggests that the snowball texture gradually evolved from rim to core, thus leading to greater fractal indices of outer zones, which represent higher complexity and maturity of the evolving system, which supports a metasomatic origin of the snowball texture. Our study demonstrates that fractal analyses with the aid of a large model are effective and efficient in characterizing and understanding complex patterns of mineral grains.