Sexual dysfunction is a common problem of increasing incidence in patients with heart failure (HF). However, there is no knowledge on which demographic, clinical, and quality of life (QoL) aspects are related to difficulties in sexual activity nor on the course of sexual difficulties in patients with HF. Data on sexual difficulties were collected at 1 and 18 months after an HF hospitalization (n = 792, mean age 69 ± 12 years, 35% female, mean left ventricular ejection fraction 33% ± 14%) by the question on sexuality of the Minnesota living with HF questionnaire. Demographic and clinical factors were assessed from medical records and QoL by Minnesota living with HF questionnaire, Medical Outcome Study 36-item General Health Survey, and Ladder of Life. In total, 48% (n = 380) of the patients perceive difficulties in sexual activity at 1 month after discharge, and 70% continued to perceive this at 18 months. Furthermore, 27% of the patients without difficulties at 1 month developed them during follow-up. Living with a partner (OR 3.76, 95% CI 2.58-5.48), younger age (OR 0.96, 95% CI 0.94-0.97)), male gender (OR 3.08, 95% CI 2.10-4.43), overall well-being (OR 1.13, 95% CI 1.00-1.27), and physical (OR 1.06, 95% CI 1.06-1.08) and emotional (OR 1.07, 95% CI 1.03-1.10) QoL were independently associated with perceived difficulties in sexual activity. Perceived difficulties in sexual activity are common in patients with HF, particularly in younger and male patients and continue over time. Patients who perceive difficulties in sexual activity report a significant lower QoL and overall well-being than those who do not.