Caloric restriction (CR) is known to extend lifespan across different species and holds great promise for preventing human age-onset pathologies. However, two major challenges exist. First, despite extensive research, the mechanisms of lifespan extension in response to CR remain elusive. Second, genetic differences causing variations in response to CR and genetic factors contributing to variability of CR response on lifespan are largely unknown. Here, we took advantage of natural genetic variation across 46 diploid wild yeast isolates of Saccharomyces species and the lifespan variation under CR conditions to uncover the molecular factors associated with CR response types. We identified genes and metabolic pathways differentially regulated in CR-responsive versus non-responsive strains. Our analysis revealed that altered mitochondrial function and activation of GCN4-mediated environmental stress response are inevitably linked to lifespan variation in response to CR and a unique mitochondrial metabolite might be utilized as a predictive marker for CR response rate. In sum, our data suggests that the effects of CR on longevity may not be universal, even among the closely related species or strains of a single species. Since mitochondrial-mediated signaling pathways are evolutionarily conserved, the dissection of related genetic pathways will be relevant to understanding the mechanism by which CR elicits its longevity effect.