To present results of patients treated on a prospective dose escalation trial of MRI-based endoluminal high dose rate (HDR) brachytherapy (BT) and concurrent chemotherapy utilizing a novel double-balloon applicator for recurrent or inoperable rectal and anal cancer. A total of 15 patients were enrolled on a prospective, institutional review board-approved dose escalation protocol evaluating endoluminal HDR-BT with concurrent chemotherapy. Inclusion criteria were histologically confirmed locally residual or recurrent cancer of the rectum or anus and prior pelvic external beam radiation therapy (EBRT). BT was delivered with a novel anorectal applicator consisting of an inner balloon, which supported 8 channels for the radioactive source, and a compliant outer balloon for optimal deformation against exophytic lesions. Applicator insertion and treatment delivery were performed under general anesthesia in 3 weekly sessions. MRI-based treatment planning was performed while under anesthesia during the first session only. Capecitabine (825 mg/m2 BID) was administered Monday-Friday on the weeks of BT. Efficacy and toxicity were evaluated by clinical assessment and MRI examinations at pre-defined intervals (3, 6, and 12 months for the first year) after the procedure. From 1/2015 to 4/2018 15 patients at a median age of 65 years (43-86) with recurrent or residual cancer (9 rectal, 6 anal) were enrolled and treated at the initial dose level of 15 Gy in 3 fractions (n = 6), the intermediate dose level of 18 Gy in 3 fractions (n = 3), and the highest dose level of 21 Gy in 3 fractions (n = 6). Treatment was delivered as planned for 14 patients; 1 patient was treated with a single-channel Bougie applicator for the third fraction due to the development of severe circumferential narrowing that prevented insertion of the endorectal applicator. On first MRI imaging post-BT, 7 patients had a complete response, 5 patients had a partial response, 1 patient had progressive disease, and 2 patients had indeterminate imaging. At a median follow up of 26 months (7-88), 10 patients (6 rectal and 4 anal) have developed a local recurrence (2 also with distant disease) which were treated with surgery (4), systemic therapy (1), or no known therapy (3). Acute (<6 months post-RT) grade 2 and 3 toxicities were observed in 1 (rectal bleeding) and 2 (anorectal pain) patients, respectively, and the only late grade 3 toxicity observed was rectal bleeding in 1 patient. Endoluminal HDR BT with MRI-based treatment planning and a novel double-balloon applicator was feasible up to a dose level of 21Gy in 3 fractions in patients with non-operable rectal or anal cancers and history of prior EBRT. The clinical efficacy and toxicity associated with this treatment should be more clearly defined with analyses of larger cohorts of patients.