A low bit rates image compression framework based on adaptive block downsampling and super resolution (SR) was presented. At the encoder side, the downsampling mode and quantization mode of each 16 × 16 macroblock are determined adaptively using the ratio distortion optimization method, then the downsampled macroblocks are compressed by the standard JPEG. At the decoder side, the sparse representation-based SR algorithm is applied to recover full resolution macroblocks from decoded blocks. The experimental results show that the proposed framework outperforms the standard JPEG and the state-of-the-art downsampling-based compression methods in terms of both subjective and objective comparisons. Specifically, the peak signal-to-noise ratio gain of the proposed framework over JPEG reaches up to 2 to 4 dB at low bit rates, and the critical bit rate to JPEG is raised to about 2.3 bits per pixel. Moreover, the proposed framework can be extended to other block-based compression schemes.