Insect herbivores can inflict substantial costs on plant reproductive success. Seed herbivory impacts directly by reducing the number of seeds and therefore the dispersal and reproductive potential of the plant. Fig trees, Ficus, provide keystone resources for tropical forests. The pollinating fig wasps develop inside figs, so consumption of unripe figs results in trees not only losing seeds but also their pollen dispersers. Selection to defend figs should therefore be strong. Seed herbivory is understudied in tropical forests and most data has been collected from fallen fruits. Here we use canopy sampling to identify fig-consuming larvae in central Panama and quantify both their consequences for the fig trees’ reproductive success and the defensive value that ants provide against these larvae. Field surveys of 46 crops from nine fig species revealed that larvae could destroy up to 80% of figs on a tree. From seven Ficus species we barcoded (using COI) 51 individual fig consuming larvae (mainly Lepidoptera) that grouped into seven molecular operational taxonomic units. Lepidopteran larvae formed two feeding strategies, either stationary within a fig or tunneling between figs. Within the context of our study, stationary larvae were specialists whereas tunneling larvae were either specialists or generalists found on different Ficus species. Trees with ants had significantly fewer figs consumed by larvae (9% ± 17% (mean ± SD) for trees with azteca ants and 16% ± 24% for trees with other ants) than did trees without ants (51% ± 27%). Our results corroborate earlier findings that hosting ants can be an effective defensive mechanism for trees against seed herbivores or other antagonistic insects. Our study contributes to a wider body of research around the networks of insects associated with figs that highlights the importance of a multitrophic approach for understanding mutualism stability and persistence in the face of antagonism.