Abstract Background Identification of apparently healthy individuals at high cardiovascular risk who may benefit from targeted preventive strategies remains challenging. Predicted heart age, the chronological age of individuals with similar absolute cardiovascular disease (CVD) risk but with no risk factors, has been proposed to appropriately describe the absolute CVD risk over conventional prediction tools. Purpose To develop multi-modality based deep learning predictors for heart age in UK-Biobank individuals. Methods We used deep learning to develop a heart age predictor based on videos from heart magnetic resonance imaging (MRI) (anatomical dimension), electrocardiograms (ECG) (electrical dimension) or both from 45000 individuals of the UK Biobank cohort (age range 45-81 years) (Figure). We estimated the heritability of heart aging and identified single-nucleotide polymorphisms associated with accelerated heart aging. We performed X-wide association study to identify non-genetic factors potentially associated with accelerated heart aging. Results We predicted age with a mean absolute error (MAE) of 2.5±0.03 years (R2=85.6±0.6%) and found that accelerated heart aging is heritable at more than 35%. MRI-based anatomical features predicted age better than ECG-based electro-physiological features (MAE=2.29±0.03 years versus 4.90±.0.08 years), and heart anatomical and electrical aging were weakly correlated (Pearson correlation=0.249±0.002). Our attention maps highlighted the aorta, the mitral valve, and the interventricular septum as key anatomical features driving heart age prediction in videos that capture the entire heartbeat cycle. We found accelerated anatomical and electrical heart aging to be genetically correlated (Pearson correlation=0.508±0.089). The most significant locus was in TTN (Titin) and it was associated with heart anatomical aging. Cardiovascular diseases, general health status, and mental health disorders were highly associated with accelerated heart aging. Conclusions A multi-modality based deep learning approach was highly predictive of heart age. Accelerated heart age has a complex biological basis with genetic and environmental correlates. Leveraging of the described pipeline to predict survival and the onset of age-related cardiovascular diseases may shed light on heart aging role in health outcomes and effectively guide personalized preventive strategies.