Image quality assessment (IQA) plays a significant role in computer vision. The performances of traditional approaches are prone to be affected by noise or complicated background. Image quality assessment towards vehicle is important for vehicle recognition and classification, which is widely applied in video surveillance. In this paper, we propose a novel full-reference image quality assessment (FR-IQA) towards vehicle scene. More specifically, we first leverage background subtraction to obtain foreground pixels, which represents the vehicle silhouette. Subsequently, we train a classifier based on Histograms of Orientation Gradient (HOG) feature to further select candidate objects from the foreground pixels. Afterwards, we design a two-stage CNN architecture to extract deep representation of reference and test images, and gradient information is also considered as the second feature. Finally, we design FR-IQA based on feature similarity index, which is utilized to recognize the similarity between reference vehicle image and test image. Extensive experiment has demonstrated that our method is effect and robust.