Due to cluttered backgrounds, long-range imaging, rapid relative motion, etc., the detection of small aerial targets remains a challenge. Existing works have illustrated the importance of temporal cues for robust small aerial target detection. However, the strong assumptions about the temporal cues give such methods high computational complexity and weak adaptability. To address this problem, we treat trajectory verification as a classification problem using convolutional neural networks. A novel small aerial target detection method that uses trajectory hypothesis and verification is proposed in this study. First, the IMU assisted inter-frame difference is used to extract target candidates from each single image. Then, based on the continuous and smoothness characteristics of the target trajectory, hypotheses of the required length are generated by linking target candidates in adjacent frames. Finally, the trajectory hypotheses are sent to a trained classification convolution neural network for verification. The detected targets are traced back from the verified trajectory. Furthermore, trajectory merge is conducted to link adjacent trajectory segments. The proposed method can achieve real-time processing. Experimental results on public datasets show that the proposed method performs better than existing methods.