The mainstream basin of the Tarim River serves as a vital ecological security barrier that prevents the merging and expansion of deserts and an important strategic corridor directly linking Qinghai and Xinjiang. With society’s development and climate change, ecological issues such as river interruption, vegetation degradation, and land desertification in the basin have notably intensified, and the ecological security is facing a critical test. Exploring the characteristics of landscape changes and their driving factors within the basin is crucial in improving the ecological environment system’s management. Based on land use data from 1980 to 2020, this study analyzed the characteristics of the spatiotemporal changes and pattern evolution of the landscape through a landscape transfer matrix and landscape pattern indices. It further revealed the impact factors of the landscape pattern through canonical correspondence analysis. The results showed that (1) in 1980–2020, the areas of desert, forest, farmland, and settlement landscapes increased, while the area of grassland landscape decreased, and the water landscape showed an “increasing–decreasing–recovery” pattern. The landscape transition types mainly included the transition from grassland to desert; mutual transitions among farmland, grassland, and forests; mutual transitions between water and grassland; and the transition from farmland to settlements. (2) The overall landscape pattern demonstrated increased fragmentation, shape complexity, and evenness with decreased aggregation. Furthermore, different landscapes exhibited distinct characteristics of landscape pattern changes; for instance, grassland landscape showed severe fragmentation, while desert landscape displayed the strongest dominance. (3) The landscape pattern was a result of the combined impact of natural and human factors, with the soil thickness (SOT), road density (ROD), annual actual evapotranspiration (AAE), population density (POD), and mean annual temperature (MAT) exhibiting significant influences. Specifically, the settlement and farmland landscapes were mainly influenced by the mean annual relative humidity (MAH), POD, GDP density (GDP), and distance to artificial water (DAW); the forest, grassland, and water landscapes were mainly influenced by the SOT, soil organic matter content (SOM), AAE, ROD, elevation (ELE), MAT, slope (SLP), and distance to natural water (DNW); and the desert landscape was mainly influenced by the DAW, DNW, SLP, AAE, SOT, SOM, and ROD. These findings can provide a scientific reference for landscape management and restoration, as well as sustainable social and economic development, in the mainstream basin of the Tarim River.