In semi-arid ecosystems where lacustrine sediments are rare, bio-proxies preserved in fluvial deposits are needed to understand environmental changes. In this study, we evaluate the potential of phytoliths as a bio-proxy in the Yamé River’s deposits at Ounjougou (Mali, West Africa) covering the middle to late Holocene (7790–4000 cal. BP). In soils, phytolith assemblages result mainly from decomposition of local vegetation but in alluvial deposits, the taphonomy of phytoliths is more complex, depending on the type of transport and deposition. In order to define the spatial origin of the phytolith assemblages, either from local (valley floor) or regional vegetation (catchment area), we took the sedimentary characteristics of the deposits into account. Using a combination of general and index approaches, phytoliths of 14 modern and 29 fossil samples from the Yamé valley were analyzed. The predominant source area of represented phytoliths varies with the fluvial energy of transport. Channel deposits, carried during periods of strong fluvial activity, contain higher numbers of savanna grass phytoliths and display a lower ratio of dicotyledon versus Poaceae phytoliths (D/P) than those deposited in the floodplain where phytoliths from the local gallery forest are more strongly represented. From the 5th millennium BP onwards, higher percentages of grass short cell phytoliths (GSCP) and lower D/P values point to gradual vegetation opening due to increasing aridity. High amounts of burned phytoliths show regular fire incidence in the gallery forest and attest for human impact on the landscapes of Ounjougou from the 7th millennium BP onwards. After 4500 cal. BP, there is evidence for pearl millet cultivation.