Heart rate variability (HRV) is a widely used marker of cardiac autonomic nervous activity (CANA). Changes in HRV with exercise training have often been interpreted as increases in vagal activity. HRV is strongly associated with heart rate, which in turn, is associated with heart size. There is strong evidence from basic studies that lower heart rate in response to exercise training is caused by morphological and electrical remodeling of the heart. In a cross-sectional study in participants of a 10 mile race, we investigated the influence of endurance exercise on HRV parameters independently of heart size and heart rate. One-hundred-and-seventy-two runners (52 females and 120 males) ranging from novice runners with a first participation to an endurance event to highly trained runners, with up to 15 h of training per week, were included in the analysis. R-R intervals were recorded by electrocardiography over 24 h. Left ventricular end diastolic volume indexed to body surface area (LVEDVI) was assessed by transthoracic echocardiography and peak oxygen consumption (VO2peak) by cardiopulmonary exercise testing. Exercise was quantified by VO2peak, training volume, and race performance. HRV was determined during deep sleep. HRV markers of vagal activity were moderately associated with exercise variables (standardized β = 0.28–0.40, all p < 0.01). These associations disappeared when controlling for heart rate and LVEDVI. Due to the intrinsic association between heart rate and HRV, conclusions based on HRV parameters do not necessarily reflect differences in CANA. Based on current evidence, we discourage the use of HRV as a marker of CANA when measuring the effect of chronic exercise.