In this paper, we proposed an estimation algorithm of the relative velocity between two driving vehicles based on CNN(Convolutional Neural Network), as the perception of the surrounding environment around an autonomous car, such as distance and speed of other vehicles, is important. The proposed algorithm estimated the velocity of a target vehicle by using a stereo camera without any other sensors. A stereo camera is a sensor that acquires simultaneously RGB images and depth maps, and it is widely used for autonomous vehicles to obtain various information. The developed CNN-based semantic segmentation model with high speed and accuracy was used to recognize the target vehicle, detecting the shape of the target object and omitting background information around it. Due to this effect, the distance between the autonomous vehicle and target vehicle is more accurately estimated. The relative speed between each vehicle was estimated using the distance and time difference between frames. The performance of the proposed algorithm was compared with conventional methods and it was confirmed that the speed of the target vehicle was more accurately estimated using only the stereo camera.