For many applications it is critical to deliver a contamination free polyethylene product. Therefore, continuous inspection of polymer production is vital. Most systems inspect a pellet stream in an at-line fashion and provide feature information about any present contaminant such as color and size. However, in certain scenarios it is also of interest to determine whether the contamination is free flowing in the pellet stream (loose) or incorporated into the polymer pellet (embedded). Typical analytical equipment does not provide this information and the classification is thus a manual and subjective task.To automate this classification, a multi-class classifier was built with a convolutional neural network (CNN), including InceptionV3, VGG16, and ResNet50, were tested with and without image augmentation and the trained model was able to achieve an accuracy greater than 95% for all classes on the test set. This is a successful application of deep learning techniques to directly improve manufacturing efficiency, and the model is currently in use for daily decision making.