Latent fingerprint enhancement is paramount as it dramatically influences matching accuracy. This process is often challenging due to varying structured noise and background patterns. The prints may be of arbitrary sizes and scales with a high degree of occlusion. There is a need for creating an end-to-end system that handles different conditions reliably to streamline this often lengthy and tricky process. In this work, we propose a Generative Adversarial Network (GAN) based architecture that effectively captures multi-scale context using Atrous Spatial Pyramid Pooling (ASPP). We have trained the network on a synthetically generated dataset, carefully designed to represent real-world latent prints. By avoiding the reconstruction of spurious ridges and only enhancing valid ridges, we avoid the generation of false minutiae, leading to better matching performance. We obtained state-of-the-art results in Sensor to Latent matching using the IIITD MOLF and Latent to Latent Matching using IIITD Latent datasets.