Loss of treated water through buried pipeline leakage is one of the pressing challenges faced by water utilities across the world. Many current pipeline inspection techniques are ad-hoc in nature and only provide a snapshot status of the system. Although a few embedded sensor-based systems offer promise for continuous monitoring, they are limited in their ability to detect leakages of all magnitudes. Lack of a comprehensive understanding of the pipeline system's structural dynamics and the correlation to leakage detection is a primary challenge. To overcome this challenge, a deep learning algorithm that uses scalogram images of vibration signals collected from accelerometers attached to the pipeline surface is developed and validated in this study. The advantage is that the complex structural features of the water pipeline systems need not be modeled if sufficient leak-based vibration signal data is available for training purposes. A convolutional neural network (CNN) model adapted from a pre-trained AlexNet network is leveraged to demonstrate the proposed approach using vibration data collected from an experimental pipeline test bed. The CNN model is found to be able to detect leakages on polyvinyl chloride (PVC) pipelines with up to 95% accuracy. A maximum accuracy of 98% is observed with acceleration signal data collected from carefully selected locations. The approach presented in this study can be adapted for effective leakage detection in real world scenarios with minimum human intervention.