Third-party threats, such as construction activities and man-made sabotage, have become the main cause of pipeline accidents in recent years. This article proposes a surveillance system for protecting the buried municipal pipelines from third-party damage based on distributed fiber optic sensing and convolutional neural network (CNN). Due to the ability of detecting very small perturbation, the phase-sensitive optical time-domain reflectometry (φ-OTDR) is employed for distributed vibration measurements along the pipelines. A two-layer classifier based on CNN is developed: one layer is used to discriminate the third-party activities from the environmental disturbance; the other is to determine the specific type of the third-party events. Meanwhile, a time-space matrix is introduced to reduce the false alarm and correct possible errors by taking into account the continuity of the signals in time and space. Field tests are carried out to validate the effectiveness of the proposed surveillance system. The recognition results show that the CNN-based classifiers achieve the accuracy of over 97%, which is 14.8% higher than that of the traditional feature-based machine learning method using random forest (RF) algorithm. It also indicates that the time-space matrix can dramatically reduce the false alarm and enhance the recognition accuracy.