Inverse lithography technology (ILT) is a leading-edge method for improving the resolution and image fidelity of optical lithography systems in semiconductor manufacturing. However, the massive computational complexity of ILT presents the inherent runtime bottleneck, which hinders its application to the large-scale lithography layouts. This paper innovates a fast ILT method by reducing both of the inherent algorithmic complexity and the required iteration number. Based on the diffraction-limited nature of optical lithography system, a bandwidth-aware acceleration approach is applied to reduce the calculation complexities of the forward imaging model and inverse gradient computation in each ILT iteration. In addition, a carefully designed algorithm based on the Nesterov accelerated gradient (NAG) is developed to achieve nearly quadratic convergence rate, thereby reducing the total number of iterations required to obtain the optimal solution. Numerical experiments show that the proposed method can improve the speed by hundreds of times over the traditional gradient-based ILT algorithms without sacrificing accuracy.