A study on microhabitat, altitudinal and seasonal influences on small mammal abundance in Mount Rungwe Nature Forest Reserve, Tanzania was carried out from March 2019 to February 2020 during the wet and dry seasons. Live traps were used in six grids and six transect lines for capturing small mammals at the low, mid, and high elevations (1700–2600 m.a.s.l.). Generalized linear models were used to examine the effects of microhabitat characteristics, altitude, and seasons on the relative abundance of small mammals. A total of 444 rodents and shrews were recorded on 4320 trap nights. Rodent species recorded included Beamys hindei, Cricetomys ansorgei, Dendromus insignis, Grammomys ibeanus, Graphiurus murinus, Lophuromys machangui, Praomys delectorum, and one shrew, Crocidura sp . Overall, P. delectorum was the most dominant species in all elevations, with 68.9% of all captures. Mid elevation had a higher abundance of small mammals (Estimate ± SE = 1.17 ± 0.49, Z = 2.37, p = 0.0176). Species abundance was influenced differently by elevation. While P. delectorum decreases with increasing elevation, L. machangui increases with an increase in elevation. Although overall small mammal abundance was not affected by microhabitat variables (Estimate ± SE = −0.08 ± 0.13, Z = −0.67, p = 0.5050), individual species ( P. delectorum and L. machangui ; Estimate ± SE = 0.13 ± 0.06, Z = 1.95, p = 0.05 and −0.31 ± 0.09, Z = 3.34, p = 0.0008 respectively) were affected differently. While P. delectorum abundance showed a positive correlation and increased with an increase in PCA1 in GLM, L. machangui had a negative trend that shows they were affected differently. Small mammal abundance, on the other hand, was affected by season and, in particular, rain, with low abundance during heavy rains and dry cold periods (Estimate ± SE = 0.51 ± 0.14, Z = 3.6, p = 0.0003 and 0.3 ± 0.15, Z = 2.08 p = 0.04 respectively). In general, the results show that microhabitat parameters, elevation, and season influenced small mammals’ abundance in MRFNR. This outcome indicates that altering the microhabitat could have an impact on the small mammal assemblage and particularly their abundance. Thus, microhabitat, elevation, and season influence small mammal abundance and can be used as a proxy for evaluating the biodiversity of montane tropical small mammal communities.