Abstract In Jakarta, the issue of river pollution due to indiscriminate waste disposal poses serious environmental and safety concerns, often leading to flooding during the rainy season. Manual surveillance by human resources has proven ineffective in addressing the escalating scale of the problem. This study presents an automated waste detection system for river surveillance in Jakarta, especially inorganic waste, utilizing video processing techniques, specifically background subtraction and frame differencing. We collected and analyzed video data from 13 rivers, including the Ciliwung, Angke, and Pesanggrahan rivers, during October and November 2023. The system’s performance was evaluated based on its accuracy in detecting waste objects, with detection rates varying significantly across different rivers. High detection accuracies were achieved in the Cipinang (93%) and Malang (90%) rivers, while lower accuracies were noted in rivers like Grogol (25%) and Cakung (17%). The overall average detection rate was 60%. These results highlight the system’s strengths in cleaner, less dynamic environments and its challenges in more complex conditions. Future research should focus on enhancing algorithm robustness, incorporating adaptive thresholding, and integrating multi-sensor data to improve detection accuracy.