Radiation necrosis (RN) is a possible late complication of stereotactic radiosurgery (SRS), but only a few risk factors are known. The aim of this study was to assess tumor location in correlation to the development of radiation necrosis for skull base (SB) and non-skull base tumors. All patients treated with radiosurgery for benign neoplasms (2004-2020) were retrospectively evaluated. The clinical, imaging and medication data were obtained and the largest axial tumor diameter was determined using MRI scans in T1-weighted imaging with gadolinium. The diagnosis of RN was established using imaging parameters. Patients with tumors located at the skull base were compared to patients with tumors in non-skull base locations. 205 patients could be included. Overall, 157 tumors (76.6%) were located at the SB and compared to 48 (23.4%) non-SB tumors. Among SB tumors, the most common were vestibular schwannomas (125 cases) and meningiomas (21 cases). In total, 32 (15.6%) patients developed RN after a median of 10 (IqR 5-12) months. Moreover, 62 patients (30.2%) had already undergone at least one surgical resection. In multivariate Cox regression, SB tumors showed a significantly lower risk of radiation necrosis with a Hazard Ratio (HR) of 0.252, p < 0.001, independently of the applied radiation dose. Furthermore, higher radiation doses had a significant impact on the occurrence of RN (HR 1.372, p = 0.002). The risk for the development of RN for SB tumors appears to be low but should not be underestimated. No difference was found between recurrent tumors and newly diagnosed tumors, which may support the value of radiosurgical treatment for patients with recurrent SB tumors.