The aim of this paper is to track objects in a sequence of surveillance video frames by providing accurate object contours. Video analysis is performed in four key phases: (i) modeling the object/background by means of the Gaussian Mixture Model and the Kalman Filter, (ii) extracting a rough contou r for the object through separating the moving objects from the still background, (iii) finding the corresponding points and the object location using the shape context matching algorithm, and eventually, (iv) extracting the exact shape contour. The suggested methodology is accurate enough to track objects in terms of scale change, translation, rotation, and partial occlusion and it can be used in real-time applications. Results on different video sequences establish the capability of this technique for a variety of applications such as traffic control cameras, speedometers, CCTV security cameras, photography and etc.