In this paper, we propose a new algorithm to detect rear-approaching vehicle using frame structure similarity based on deep learning algorithm for use in agricultural machinery systems. The commonly used deep learning models well detect various types of vehicles and detect the shapes of vehicles from various camera angles. However, since the vehicle detection system for agricultural machinery needs to detect only a vehicle approaching from the rear, when a general deep learning model is used, a false positive is generated by a vehicle running on the opposite side (passing vehicle). In this paper, first, we use Faster R-CNN model that shows excellent accuracy rate in deep learning for vehicle detection. Second, we proposed an algorithm that uses the structural similarity and the root mean square comparison method for the region of interest(vehicles area) which is detected by Faster R-CNN between the coming vehicle and the passing vehicle. Experimental results show that the proposed method has a detection rate of 98.2% and reduced the false positive values, which is superior to general deep learning method.