Abstract Volumetric measurements of whole tumor and its components on MRI scans, facilitated by automatic segmentation tools, are essential to reduce inter-observer variability in monitoring tumor progression and response assessment for pediatric brain tumors. Here, we present a fully automatic segmentation model based on deep learning that reliably delineates the tumor components recommended by the Response Assessment in Pediatric Neuro-Oncology (RAPNO) working group for evaluation of treatment response. Multi-parametric MRI (mpMRI) scans (T1-pre, T1-post, T2, and T2-FLAIR), acquired on multiple MRI scanners with different field strengths and vendors, for a cohort of 218 pediatric patients with a variety of histologically confirmed brain tumor subtypes were collected. The mpMRI scans were co-registered and manually segmented by experienced neuroradiologists in consensus to identify the tumor subregions including the enhancing tumor (ET), non-enhancing tumor (NET), cystic components (CC), and peritumoral edema (ED) regions. A convolutional neural network model based on DeepMedic architecture was trained using mpMRI scans as the inputs for segmentation of the whole tumor and subregions. The trained model showed excellent performance in segmentation of the whole tumor, as suggested by median dice of 0.90/0.85 for validation (n = 44)/independent test (n = 22) sets. ET and non-enhancing components (union of NET, CC, and ED) were segmented with median dice scores of 0.78/0.84 and 0.76/0.74 for validation/test sets, respectively. The automated and manual segmentations demonstrated strong agreement in estimating VASARI (Visually AcceSAble Rembrandt Images) MRI features with Pearson’s correlation coefficient R > 0.75 (p < 0.0001) for ET, NET, CC, and ED components. Our proposed automated segmentation method developed based on MRI scans acquired with different protocols, equipment, and from a variety of brain tumor subtypes, shows potential application for reliable and generalizable volumetric measurements which can be used for treatment response assessment in clinical trials.