Forest fragmentation often leads to changes in the abundance and composition of various organisms, thereby affecting biotic interactions. Currently, many studies lack consensus on the magnitude or direction of the impact of forest fragmentation on plant-animal interactions. Here, we employed seed-rodent interaction as a case study and assessed the removal by rodents of 66,960 seeds belonging to 31 woody species in 18 fragmented forests with areas of 1.05–14,517.63 ha, in a seriously fragmented tropical area in Southwest China for two consecutive years. Our study provided strong evidence that forest fragmentation affected seed-rodent interactions but did not follow a consistent pattern between years. Forest size and edge effects on seed removal were mainly due to variations in seed availability but not rodent activity. Interestingly, the effect of fragment size on seed removal was species-specific with monotonic decreases as well as U-shaped and humped-shaped patterns. These remarkable spatiotemporal and interspecific variations pertaining to forest fragmentation impact on seed removal may translate into subsequent seed germination and seedling regeneration processes, which may further influence the species composition in different fragmented forests. This complexity in responses among plant species reiterates the need to consider the interspecies specificity in conservation and management initiatives of fragmented forests. Additionally, we recommend that multiple field surveys involving multiple species be conducted in future studies focusing on forest fragmentation to help us better understand fragmentation effects on ecological processes.