A material recovery facility (MRF) can transform municipal solid waste (MSW) into a valued commodity called refuse-derived fuel (RDF) as a promising solution to waste-to-energy conversion. The quality of the produced RDF significantly relies on the composition of in-feed waste and waste characterization method applied for auditing purposes, a process that is both time-consuming and fraught with potential hazards. This study focuses to enhance the workflow of the waste characterization process at an MRF. A solution named Smart Sight is proposed to detect and classify waste based on videos recorded after processing MSW through a mechanical sorting line consisting of bag breakers and trommel screens. A comprehensive dataset is created encompassing thirteen mixed waste classes from single and multi-family streams. The dataset is preprocessed with motion compensation techniques and frame differencing methods to extract and refine valuable frames. A one-stage YOLO detector model is then trained over the dataset. The experimental results show that the proposed method works efficiently at detecting and classifying waste objects in indoor MRF environments. Accuracy, precision, recall, and F1 score related to the proposed solution are found to be 0.70, 0.762, 0.69 and 0.72, respectively, with a mAP@0.5 of 0.716. The proposed approach is validated using data collected from local MRF by comparing the estimated waste composition values of the proposed solution with laboratory results obtained through current standardized industrial practices. Comparison reveals that waste characterization estimation obtained is consistent with the laboratory results, inferring that Smart-Sight is a viable tool for estimating waste composition.