Urban public transport has become a preferred choice for alleviating traffic congestion. The bus passenger OD (origin–destination) demand prediction based on bus operational data is the key technology to realize urban intelligent transportation system. However, most of the existing bus OD demand prediction methods only considered regional passengers. The problem of the OD demand prediction based on historical OD matrices of bus lines is still not easy to implement, exceptionally, which is suitable for most of the urban bus lines. This paper presents a non-symmetric spatial-temporal network (NSTN) based on convolutional neural network (CNN) and convolutional long short-term memory (ConvLSTM) network to predict bus OD. NSTN contains the station spatial component (SSC) module and the spatial-temporal component (STC) module. SSC consists of two CNNs to learn the OD features and the DO (destination-origin) features, respectively. To make the prediction shift to the OD features, the non-symmetric input is designed. STC extracts spatial-temporal features based on ConvLSTM. Compared with other methods, NSTN has the best performance measured by symmetric mean absolute percentage error (SMAPE) and root mean square error (RMSE), where its SMAPE falls by 4.3 percentage points to 16.4 percentage points and RMSE falls by 23.1 percentage points to 69.9 percentage points. Experimental results on other bus lines show that NSTN has strong generalization ability.