Generally, more than half of smaller singular values and corresponding singular vectors should be abandoned to achieve the image compression function for the compression method based on singular value decomposition (SVD). Although these discarded parts contain some noise and fuzzy factors, they also contain some detailed information to boost image reconstruction. To overcome this problem, we present a novel lossy image compression method named singular vector sparse reconstruction (SVSR) keeping the sparse representation data of more singular vector to boost the performance of SVD based image compression method in compression ratio and reconstruction quality. Specifically, we treat the singular vector as a signal and express it sparsely through sparse sampling based on the analysis of the characteristics of the singular vector. In particular, the compression ratio of the proposed method is about 70% higher than that of the traditional SVD method. Evaluation on several image data and the experimental results with different image compression algorithms clearly demonstrate the advantages of our proposed SVSR algorithm in compression ratio and reconstruction quality.