Vision-based hardware driver assistance systems are the most important systems in the world because of their low cost and ability to provide information on driving environments. Improving safety and reducing accidents are the two main objectives of these systems. For this, in this paper a new Vision-based Hardware Advanced Driver Assistance System (VH-ADAS) based machine learning incorporating the hybridization of Support Vector Machine (SVM)-Histogram of Oriented Gradient (HOG) classifier and Particle Swarm Optimization (PSO) technique is proposed for traffic scenes from both video and captured images. First, the proposed system uses a feature extraction method based on the HOG. Then, the Particle Swarm Optimization technique is used for selection and so to optimize the features. The SVM method is applied to obtain fast detection and high accuracy. Finally, a hardware synthesizable architecture of the complete system was developed and then co-simulation validity was succeeded using the Matlab-Vivado System Generator (VSG) and a Field Programmable Gate Array (FPGA). The results show that the proposed new system supports real-time detection in both images and video. Also, they show that the proposed vehicle detection method is competitive in terms of parallel run time with only 1.483 ms and in terms of accuracy rate with only 97.84%.