Coevolutionary models suggest that herbivores drive diversification and community composition in plants. For herbivores, many questions remain regarding how plant defenses shape host choice and community structure. We addressed these questions using the tree genus Inga and its lepidopteran herbivores in the Amazon. We constructed phylogenies for both plants and insects and quantified host associations and plant defenses. We found that similarity in herbivore assemblages between Inga species was correlated with similarity in defenses. There was no correlation with phylogeny, a result consistent with our observations that the expression of defenses in Inga is independent of phylogeny. Furthermore, host defensive traits explained 40% of herbivore community similarity. Analyses at finer taxonomic scales showed that different lepidopteran clades select hosts based on different defenses, suggesting taxon-specific histories of herbivore-host plant interactions. Finally, we compared the phylogeny and defenses of Inga to phylogenies for the major lepidopteran clades. We found that closely related herbivores fed on Inga with similar defenses rather than on closely related plants. Together, these results suggest that plant defenses might be more evolutionarily labile than the herbivore traits related to host association. Hence, there is an apparent asymmetry in the evolutionary interactions between Inga and its herbivores. Although plants may evolve under selection by herbivores, we hypothesize that herbivores may not show coevolutionary adaptations, but instead "chase" hosts based on the herbivore's own traits at the time that they encounter a new host, a pattern more consistent with resource tracking than with the arms race model of coevolution.