To report health-related quality of life (HRQOL) and toxicity outcomes at 4-years in prostate cancer patients treated with high-dose rate (HDR) brachytherapy as monotherapy using single-fraction of 19 Gy or 2 fractions of 13.5 Gy (27 Gy/2 fractions one week apart). Patients with low or intermediate-risk prostate cancer (NCCN definition) were accrued to a randomized phase II clinical trial of HDR brachytherapy as monotherapy, randomized to either 19 Gy/1 or 27 Gy/2. HRQOL (Expanded Prostate Cancer Index Composite [EPIC]), urinary symptoms (International Prostate Symptom Score [IPSS]), and toxicity (Common Terminology Criteria for Adverse Events [CTCAE], v4.0) were assessed prospectively. Comparison between the two arms was analyzed using Fisher exact test. Univariate linear and logistic regression analysis was used to investigate association between HRQOL/toxicity and baseline covariates. Median follow-up is 51 months. 170 patients were treated (87 with single 19 Gy, 83 with 27 Gy/2). Late (>6 months) GI toxicity was minimal in both arms, with G-2 toxicity <1%, and no difference between the arms. Late G-2 urinary toxicity was low and <2%, except for urinary frequency and incontinence. G-2 urinary frequency was similar between arms (19% for 19Gy/1 and 10% 27Gy/2; p=0.27). There was a significantly later G-2 urinary incontinence with 19 Gy/1, and none with 27 Gy/2 (6% vs 0%, p=0.0318). G 2-3 urinary tract obstruction was only observed in 4 patients, all of whom were treated with 19 Gy/1. No change in the median EPIC scores at 4 years (follow-up - baseline score) was observed with 27 Gy/2 for urinary, bowel or hormonal domains, but a decline in sexual domain by 16 points. In the single fraction arm, there was no change in the bowel or hormonal domains too, but a drop in sexual domain by 9 points, comparable to 27 Gy/2 (p=0.53). There was a significantly greater drop in urinary domain, function, and bother with 19 Gy/1 (-3.5 vs 0.0, p=0.003; -5.0 vs 0.0, p=0.004; and -3.57 vs 0.0, p=0.006, respectively). Urinary QOL was slightly worse with 19 Gy/1 compared to 27 Gy/2, with greater clinically significant decrement (defined as a decrease in EPIC score >0.5 SD) in urinary domain (46.15% vs 16.67%, p=0.007), function (52.5% vs 15.38%, p=0.0008) and bother (43.59% vs 16.67%, p=0.0136). Treatment with 19 Gy/1 (OR 4.286, p=0.0082) and higher PTV V100 (OR 1.6, p=0.02) were the only factors predictive of clinically significant decrement in urinary QOL. There was no difference in clinically significant decrement between arms in other QOL domains (bowel, sexual, or hormonal). Both single 19 Gy and 13.5 Gy x 2 are well tolerated with low toxicity and good HRQOL at 4 years. Single fraction 19 Gy is associated with higher rates of G-2 urinary toxicity, worse urinary function and bother, and greater decline in urinary QOL at 4 years. Treatment with 19 Gy/1 and having higher PTV V100 are predictive of decrement in urinary QOL.