Many deep learning-based raindrop removal methods have been proposed recently and demonstrated good performance. However, most deep learning based raindrop removal methods struggle to restore distorted color and texture resulting from the refraction of raindrops emitted from different background points. To overcome this challenge, we propose a novel recurrent single-image deraindrop approach that utilizes luminance priors and contextual feature aggregation. Our method decomposes the deraindrop process into a raindrop detection network and a raindrop removal network, leveraging the luminance differences between the raindrop and non-raindrop regions for accurate detection. Contextual feature aggregation, achieved by dilated convolutions with different rates, helps recover raindrop attachment areas and maintain consistency in color and texture. We have extensively evaluated our method on synthetic and real raindrop datasets, demonstrating its effectiveness, generalization capability, and superior performance compared to state-of-the-art methods.