Elevated atmospheric N deposition is predicted to increase plant growth and C storage in N-limited systems, but this assumes that no mitigating ecological factors are influenced by N deposition. Many herbivores are also N-limited and so increased herbivore damage in response to N deposition may moderate, or even eliminate, gains in plant growth under elevated N. Thus the response of herbivores to N deposition could influence the onset and magnitude of N saturation resulting in expedited decreases in C storage. Since tree susceptibility to herbivores is partially under genetic control and will be influenced by site characteristics, we tested whether the interaction between N deposition, tree growth, and herbivore damage depends on tree genetic variation in susceptibility to herbivores at different locations. By monitoring 12 half-sib families of northern red oak ( Quercus rubra) saplings at two common garden sites in south-central Pennsylvania, we found that herbivores were pervasive at both sites, resulting in 13% chewing damage, 16% galling damage and 28% browsing damage. N addition significantly affected browsing damage (8% overall increase) and chewing damage (19% overall increase) but did not affect galling damage. However, this effect was strongly dependent on the Q. rubra family and location, both of which influenced oak susceptibility to herbivore damage independent of N additions. Greater browsing damage on trees under N addition caused reductions in relative height growth, and this effect also depended on tree family and location. Our result suggests that herbivory, mediated by tree genetic lineage and site-specific conditions, may represent an important component of C sinks and N saturation under elevated N deposition.