Shadow detection is a basic task of remote sensing image analysis, but it is often seriously disturbed by vegetation, water bodies, and black objects. It is observed that vegetation and dark objects often show a dark look in visible bands but brighter in the near-infrared (NIR), and is also noticed that the reflection of inland water bodies in the green band is stronger than that in the blue band. Taking advantage of these physical properties and combining them with the bluish and dark appearance of shadows, we propose a simple but effective shadow detection method for multispectral remote sensing images. These physical properties are used to create transformation models that suppress features such as vegetation, water bodies, etc., but at the same time enhance shadows. Then, we transform the shadow representation into a color space to generate candidate shadows using dominant color components. To separate shadows from the others, we propose two indexes, the normalized Color Difference Composite Index (CDCI) and Color Purity Index (CPI), and fuse them to achieve shadows and their confidence. The experimental results indicate that the proposed method can effectively detect the shadows in multispectral images and outperforms the state-of-the-art approaches.