Dysregulation of daytime cortisol activity has been associated with stress-related pathologies. Research suggests that early environmental adversity might shape cortisol activity. However, little is known about the genetic and environmental contributions to early cortisol and how this varies as a function of environmental circumstances. The goals of the study were to estimate the genetic and environmental contributions to daytime cortisol secretion in infant twins and to investigate whether these contributions varied as a function of familial adversity (FA). Participants were 517 6-month-old twins. Salivary cortisol was collected when the infants woke up at home and in the morning, upon arrival at the laboratory. Familial adversity was defined by seven perinatal and postnatal risk factors: maternal smoking during pregnancy, low birth weight, low family income, low maternal education, single parenthood, young motherhood, and maternal hostile/reactive behaviors. Genetic and environment contributions to cortisol activity were estimated for high (three risk factors or more: 21.3% of the sample) versus low FA. Genetic factors accounted for cortisol levels in different ways: a moderate "main effect" of genes was found for home-based awakening cortisol, whereas the contribution of genes to morning cortisol was conditional to FA. Genetic factors accounted for most of the variance in morning cortisol in high family adversity but not in low family adversity. Early FA modulates the heritability of morning cortisol in infants. The results are consistent with the diathesis-stress model, with genetic factors more likely to be expressed in adverse settings.