The robust detection of infrared small targets plays an important role in infrared early warning systems. However, the high-brightness interference present in the background makes it challenging. To solve this problem, we propose a weighted improved double local contrast measure (WIDLCM) algorithm in this paper. Firstly, we utilize a fixed-scale three-layer window to compute the double neighborhood gray difference to screen candidate target pixels and estimate the target size. Then, according to the size information of each candidate target pixel, an improved double local contrast measure (IDLCM) based on the gray difference is designed to enhance the target and suppress the background. Next, considering the structural characteristics of the target edge, we propose the variance-based weighting coefficient to eliminate clutter further. Finally, the targets are detected by an adaptive threshold. Extensive experimental results demonstrate that our method outperforms several state-of-the-art methods.