Mangroves play a vital role in protecting the coastal community from the climate change effect and in the restoration of the coastal ecosystem. This research has been designed to determine the spatial and seasonal changes of potentially toxic elements’ (PTEs) concentration in sediments and their potential source contribution among the different human-driven processes in Sundarbans, Bangladesh. Different pollution evaluation indices, random forest (RF) model, conditional inference tree (CIT), self-organizing map (SOM), geographical information system (GIS), and principal component analysis (PCA) were used for the interpretation of sources and risk assessment of PTEs. The mean concentration of PTEs both in winter and monsoon seasons has fallen below the threshold effect level but exceeded the rare effect level of marine sediments quality standards. Results showed that the PTEs were significantly enriched (EF > 1.00 < 70.00) in sediments, whereas the Cd enrichment (7.00% samples) was very alarming (EF = 60–70). Except for Zn and Cd, other PTEs were enriched in 30–60% samples. The highest geoaccumulation and contamination factors for Cd were observed in 46–72% of samples. The ecological risk (ER) factors showed similar results where Cd showed strong to very strong factors (ER = 110–2218) in 80% of samples. The CIT explained the natural/geogenic and anthropogenic sources of pollution, where the higher CIT values for Cd indicated industrial, aquaculture, and coal-based thermal powerplant. The RF model provided that shrimp firms, power plants, industry, and seaport were recognized as the influential sources for Zn, Pb, Cr, Cd, and As in sediments. Though Pb and As were found as the most significant pollutants, Cd was identified as a severe threat to ecology and public health. Based on CIT, RF, SOM and PCA the order of PTEs in mangroves sediment were:industrial/urban > aquaculture/shrimpfirm > powerplant > seaportoperation > tourism > geogenic/natural. The present study will help the policymakers for effective and sustainable management of the mangrove ecosystem.