The construction and demolition sector generates a substantial portion of Australia's total waste, with plastics being a key recyclable component. The perceived financial impracticality of sorting and separating waste, coupled with the simplicity of landfilling processes often contribute to mixed material loads sent directly to landfills. Therefore, developing a commercially feasible system that can accurately separate the generated waste is imperative. This paper presents a comprehensive study on using RGB images for deep learning-based construction and demolition plastic waste classification by resin type. A large and specialised dataset of end-of-life plastic waste images is gathered. This dataset comprises four commonly used plastic types in construction projects—ABS, HDPE, PS, and PVC. Leveraging Transfer Learning with models pre-trained on ImageNet, highly accurate models tailored to this classification task are developed in this paper. Advanced Convolutional Neural Network and Vision Transformer-based models, including ResNet, ResNeXt, RegNet, and Swin Transformer, are trained and evaluated on this dataset. Another contribution of this work is Knowledge Distillation from a large, computationally intensive, and accurate model to enhance the accuracy of fast and compact models specifically designed for deployment on edge devices. This study applies Knowledge Distillation by using the output class probabilities of the large, computationally intensive Swin Transformer model to enhance the accuracy of the fast and lightweight MobileNetV3 models. The results demonstrate that RGB images offer a practical alternative to other costly and complex systems for effective plastic identification, due to their availability, low cost, ease of use, simple setups, and robustness to variations in operational conditions.