The skeleton structure composed of mineral aggregates is the main body to bear and transfer external loading in asphalt mixtures. To investigate the loading transfer mechanism of the mineral aggregate skeleton, the uniaxial penetration test and Discrete Element Method (DEM) were conducted for the Mineral Aggregate Mixture (MAM) to analyze its mechanical behavior. The three-dimensional strong force chain (SFC) was identified and evaluated based on the proposed recognition criterion and evaluation indices. The results indicate that 4.75 mm should be the boundary to distinguish the coarse and fine aggregates. The skeleton composed of aggregates located on SFCs has better bearing and transferring loading capacity due to its SFC number, average length, and total length decreasing with an increase in the aggregate size. Compared to SMA-16 and OGFC-16, AC-16 exhibits a higher number and total length of its SFC, a smaller average length of its SFC, and a lower average strength of its SFC. Consequently, AC-16 has a lower bearing and transferring loading capacity than that of SMA-16 and OGFC-16. In addition, approximately 90% of SFCs can only transfer external loading downward through 3–5 aggregates. The average direction angle of the SFC formed by fine aggregates is significantly higher than those formed by coarse aggregates. This indicates that the load transfer range of MAM composed of fine aggregates is noticeably larger, leading to lower loading transfer efficiency.