In this paper, a wavelet convolutional neural network (WNN) consisting of a one-dimensional (1D) convolutional neural network and a self-adaptive wavelet neural network has been proposed and demonstrated experimentally for temperature measurement in a Brillouin optical time domain reflectometry (BOTDR) system. Based on the analysis of the system noise, it follows the Gaussian white noise distribution along the time-related sensing distance. The impact of the noise in time-domain on the measured Brillouin gain spectra (BGSs) could be neglected, so that the BGSs in the fiber can be regarded as a series of 1D input data of the proposed WNN. Different self-adaptive wavelet activation functions connected to each output of the full-connection network are adopted to realize the multi-scaled analysis and the scale translation, which can obtain more local characteristics in frequency-domain. The output extracted by the WNN is Brillouin frequency shift (BFS), which presents linearity correlation to the actual temperature. Considering the multi-parameters including different frequency ranges, signal-to-noise-ratios (SNRs), BFSs and spectral widths (SWs), a general model of the proposed WNN is trained to handle more extreme cases, in which it doesn't require retraining for different single-mode (SM) optical fibers in BOTDR sensing system. The performances of the WNN are compared with other two techniques, the Lorentzian curve fitting based on Levenberg-Marquardt (LM) algorithm and the basic neural network (NN) containing input and output layers together with two hidden layers. Both the simulated and measured results show that the WNN has better robustness and flexibility than the LM and the NN. Besides, the computational accuracy of the WNN is improved and the fluctuation of that is slighter, especially when the SNR is less than 11 dB. Moreover, the WNN takes approximately 0.54 s to measure the temperature from the 18,000 collected BGSs transmitted through the 18 km SM optical fiber. The calculating time of the WNN is greatly reduced by three orders of magnitude in comparison with that of the LM, and is comparable to that of the NN. It proves that the proposed WNN may provide a feasible or even better scheme for the robust and fast temperature measurement in BOTDR system.