Geomorphological dynamics is the dominating process behind the Sundarbans mangrove forest development. Here, satellite images can illuminate Sundarbans’ erosion–accretion dynamics to improve our understanding of better river management. Previous morphological studies were on mangrove vegetation loss or gain and shoreline changes detection. Nevertheless, those studies did not provide channel/river specific erosion and accretion dynamics for the Sundarbans in Bangladesh. In this study, we analyzed historical Landsat satellite images from 1980 to 2019 for six major rivers, namely Passur, Shibsa, Arpangasia, Baleshwar, Malancha, and Shella of the Sundarbans in Bangladesh, which carry freshwater from upstream and supply them into all the tributaries of Sundarbans. Image analysis shows a remarkable erosion rate in all the major rivers. The erosion rates were significantly higher for Passur (2139 ha, 55 ha/yr), Baleshwar (2122 ha, 54 ha/yr), and Shibsa (1809, 46.5 ha/yr) rivers compared to the other three rivers (Arpangasia-995 ha, 26 ha/yr; Malancha-774 ha, 20 ha/yr; Shella-332 ha, 8.5 ha/yr). The spatial pattern of lateral erosion was not uniform for all major rivers. The southern parts of all the rivers were prone to severe erosion compared to the central and northern parts. Reduced freshwater flow has less effect on the lateral erosion of major rivers of the Sundarbans. Besides, NDVI analysis shows that vegetation density has declined tremendously over the last 40 years (highest NDVI value in 1980 and 2019 was 0.87 and 0.44 respectively) along the major river banks as well as in whole Sundarbans which in turn reduces soil binding capacity. Hence, our work suggests that mangrove restoration with careful consideration of biophysical characteristics in the degraded site will aid better resistance against strong wave action.