Agricultural soil contamination is increasing day-by-day and becoming a major problem over the globe. Trace elements accumulation in the bulk soil is frequently documented, however, there is no precise mechanism of their distribution in the different soil aggregates level. We collected twelve composite soil samples from banana fields, fallow land, rice cultivated with pond water (rice field-I), and rice cultivated with rain water (rice field-II). We separated soil samples into four different size of aggregates (4-2, 2–0.25, 0.25–0.053, <0.053-mm) and then, aggregate stability (MWD), soil organic carbon (SOC), and heavy metals content (Pb, Cd, Cr, As, Fe, Mn, Zn, Ni, Co, Cu) in the soil samples were measured with different techniques. Results showed that MWD was higher in the rice-based land use, which was significantly contributed by SOC (p < 0.001). The concentration of Pb, As, Cd, Fe, and Mn were increased, while Cu and Zn concentration were reduced with increasing aggregate size (p < 0.05). In contrast, aggregate size did not influence on Ni and Co accumulation (p > 0.05). Moreover, macroaggregate acted as an accumulator for Fe, Mn, and As, while all the aggregate fractions acted as accumulators for Cu and Zn. Our study indicated that MWD, SOC, aggregate size and composition, and metal species were the controlling factors of trace elements accumulation and distribution in the various sizes of soil aggregates.