(1) Background: Isolated hypogonadotropic hypogonadism (IHH) is a genetic condition characterized by impaired puberty and fertility. IHH can significantly impact patient health-related quality of life (HRQoL), sexual satisfaction (SS) and mood. (2) Methods: Participants included 132 IHH subjects (89 men and 43 women) and 132 sex- and age-matched controls. HRQoL, depressive symptoms, erectile dysfunction (ED), and SS were assessed in an online survey using the Zung Self-Rating Depression Scale (SDS), 15D instrument of HRQoL (15D), Sexual Satisfaction Questionnaire (SSQ), and 5-item International Index of Erectile Function (IIEF-5). (3) Results: QoL and SS were significantly lower in the IHH group vs. controls. There was a high rate of ED (53.2% vs. 33%, p = 0.008) and depressive symptoms (45.00 ± 17.00 vs. 32.00 ± 12.00, p < 0.001) in patients vs. controls. The age of patients at IHH diagnosis inversely correlated with their overall 15D scores. An alarming non-compliance rate was seen (51.6%). No differences were found between scores of patients receiving hormone replacement therapy (HRT) and untreated subjects in any of the scales. (4) Conclusions: The HRQoL, SS, ED, and depression levels observed in IHH patients, despite HRT, are alarming. Late IHH diagnosis may have a particularly negative impact on HRQoL. More attention should be devoted to HRT adherence and various HRQoL aspects of IHH patients.