Conventional saliency detection algorithms usually achieve good detection performance at the cost of high computational complexity, and most of them focus on visible images. In this paper, we propose a simple and effective saliency detection framework, which can adapt to the characteristics of visible or infrared images. The proposed approach can be seen a three-step solution. On the first step block-based image compressed reconstruction is applied to the input image for reducing the computational complexity. On a second step a local contrast technique is used at the block level to obtain a primary saliency map. In this step, the appropriate features such as color or intensity will be selected for different kinds of input images. Finally, the last step uses a linear combination of feature coefficients to refine the salient regions from the primary saliency map so as to generate the final saliency map. The experimental results show that the proposed method has desirable detection performance in terms of accuracy and runtime.