Electroencephalographic (EEG) source localization is a fundamental tool for clinical diagnoses and brain-computer interfaces. We investigate the impact of model complexity on reconstruction accuracy by comparing the widely used three-layer boundary element method (BEM) as an inverse method against a five-layer BEM accelerated by the fast multipole method (BEM-FMM) and coupled with adaptive mesh refinement (AMR) as forward solver. Modern BEM-FMM with AMR can solve high-resolution multi-tissue models efficiently and accurately. We generated noiseless 256-channel EEG data from 15 subjects in the Connectome Young Adult dataset, using four anatomically relevant dipole positions, three conductivity sets, and two head segmentations; we mapped localization errors across the entire grey matter from 4,000 dipole positions. The average location error among our four selected dipoles is ∼5mm (±2mm) with an orientation error of ∼12∘ (±7∘). The average source localization error across the entire grey matter is ∼9mm (±4mm), with a tendency for smaller errors on the occipital lobe. Our findings indicate that while three-layer models are robust under noiseless conditions, substantial localization errors (10–20mm) are common. Therefore, models of five or more layers may be needed for accurate source reconstruction in critical applications involving noisy EEG data.