Outdoor images are often degraded by haze and other inclement weather conditions, which affect both consumer photographs and computer vision applications severely. Therefore, researchers have proposed plenty of restoration approaches to deal with this problem. However, it is hard to tackle the color distortion problem in restored images with ignoring the differences between fog and haze. Meanwhile, the atmospheric light is also an important variable that influences the global illumination of images. In this paper, we analyze the physical meaning of atmospheric light first, and estimate atmospheric light by a novel method of obtaining the sky region in images, which is based on our newly proposed sky region prior. Then after exploring physical characteristics of fog and haze, we explain why images taken in haze appear yellowish, and eliminate this phenomenon by our adaptive channel equalization method. Quantitative comparisons with seven state-of-art algorithms on a variety of real-world haze images demonstrate that our algorithm can remove haze effectively and keep color fidelity better.