The segmentation of endocardium and epicardium of left ventricle (LV) in cardiac MR image sequences play a crucial role in clinical applications. Active shape model (ASM) based methods are often used to extract the LV boundaries with the steps of searching and representation. However, due to the challenges, such as interior papillary muscles, complicated outside tissues and weak boundaries, the searching may be partially incorrect and the representation cannot reflect the reliable part of the contour. In this paper, a feature competition based searching strategy is proposed by exploiting both the information of the object and background to reduce the error of searching. Then, we propose a partial sparse shape model to effectively represent the searched shape. This representation is able to retain the partial reliable contour while reconstructing the unreliable part approximating to the real contour. Moreover, the incremental learning algorithm is exploited to construct a patient-specific appearance model to increase the accuracy and efficiency of image sequence segmentation. Experimental results on cardiac MR image sequences demonstrate that the proposed method improves the segmentation performance and also reduces the error accumulation compared to the existing methods.