Background Low grade inflammation is of pathogenic importance in the development of cardiovascular disease (CVD) and type 2 diabetes. The inflammation marker YKL-40 correlates with insulin resistance and is highly expressed in atherosclerotic plaques. We aimed to investigate whether YKL-40 could predict overall and cardiovascular (CV) mortality in a 50+ years population without known CVD. Methods A representative population sample of 639 individuals aged 50–89 years was recruited from general practices. Examination at baseline included echocardiography and blood and urine samples for CV risk factors and markers including lipids, high sensitive C-reactive protein (hsCRP), N-terminal fragment of pro-brain natriuretic peptide (NT-proBNP) and urinary albumin/creatinine-ratio (UACR). Median follow-up period was 5.0 (0.17–5.28) years. Results In subjects without diabetes and CVD at baseline, increasing YKL-40 levels independently predicted overall and CV mortality rate with hazard ratios of 1.58 (95% confidence interval (CI), 1.12–2.23, p = 0.009) and 1.57 (95% CI, 1.00–2.46, p = 0.049) after adjustment for age, sex, smoking, total cholesterol, hsCRP, NT-proBNP and UACR. In combined Kaplan–Meier analyses, baseline values of both YKL-40 and UACR above median significantly predicted increased cumulative overall and CV mortality rates in subjects without diabetes or CVD at baseline (30.6% vs. ≤ 8%, respectively 10.6% ≤ 3%, p < 0.0001). Conclusions YKL-40 seems to be an independent predictor of overall and CV mortality in an elderly part of the general population without diabetes and CVD. YKL-40 and UACR are both independent predictors, that seem to predict overall and CV mortality in a synergistic way.