Rubble pile asteroids are widely understood to be composed of reaccumulated debris following a catastrophic collision between asteroids in the main asteroid belt, where each disruption can make a family of new asteroids. Near-Earth asteroids Ryugu and Bennu have been linked to collisional families in the main asteroid belt, but surface age analyses of each asteroid suggest these bodies are substantially younger than their putative families. Here we show, through a coupled collisional and dynamical evolution of members of these families, that neither asteroid was likely to have been created at the same time as the original family breakups, but rather are likely remnants of later disruptions of original family members, making them second, or later, generation remnants. Our model finds about 80% and 60% of asteroids currently being delivered to near-Earth orbits from the respective families of New Polana and Eulalia are second or later generation. These asteroids delivered today in the 0.5-1 km size range have median ages since their last disruption that are substantially younger than the family age, reconciling their measured crater retention ages with membership in these families.