We study the $D$-term inflation scenario with a nonperturbative K\"ahler potential of the dilaton field. Although the FI term which leads an inflationary expansion is given by the derivative of the K\"ahler potential with respect to the dilaton in heterotic string models with anomalous U(1), the too large magnitude is problematic for a viable $D$-term inflation. In this paper, we point out that the K\"ahler potential with a nonperturbative term can reduce the magnitude of FI term to desired values while both the dilaton stabilization and $D$-term domination in the potential are realized by nonperturbative superpotential.