The prevalence of physical inactivity has been rising in many countries in recent years, adding to the burden of non-communicable diseases and affecting overall health worldwide. The aim of this study was to determine the comprehensive assessment of the prognostic value of physical activity in leisure time on mortality from ischemic heart disease (IHD) by gender separately for those respondents who were diagnosed with IHD and for those who were not diagnosed with IHD in their baseline health survey. In the baseline survey (2006-2008), 7100 men and women ages 45-72 were examined within the framework of the international study Health, Alcohol, and Psychosocial Factors in Eastern Europe (HAPIEE). A total of 6770 participants were available for statistical analysis (after excluding 330 respondents due to missing information on study variables). Physical activity was determined by leisure-time physical activities (hours/week). All participants in the baseline survey were followed up for IHD mortality events until 31 December 2018. Using multivariate Cox regression analysis, it was found that moderate and higher levels of physical activity significantly reduced the risk of IHD mortality (HR = 0.54, p = 0.016 and HR = 0.60, p = 0.031, respectively) in men who were not diagnosed with IHD at baseline compared with physically inactive subjects. It was found that among men and women who were diagnosed with IHD at baseline, physical activity reduced the risk of mortality from IHD compared with those who were physically inactive (HR = 0.54, p = 0.021 and HR = 0.41, p = 0.025, respectively). Using mediation analysis, it was found that physical activity directly predicted statistically lower IHD mortality (p < 0.05) in men and women. High physical activity was a significant factor that directly predicted statistically lower IHD mortality in men, regardless of whether subjects had IHD at baseline or not. However, only moderate physical activity was a significant factor that directly predicted statistically lower IHD mortality in the women group with IHD at baseline.