This paper addresses the inherent nonlinear problem in indoor position estimation utilizing Angle of Arrival (AoA) measurements. We investigate the influence of deployment geometry on system performance through both analytical methods and Monte-Carlo simulations, shedding light on the limitations of single and multi-anchor setups and underscoring the imperative for advanced multi-anchor localization methods. In response to this problem, we propose a multi-anchor solution with outlier rejection that efficiently considers the nonlinearity of the model. For each anchor, our approach approximates the probability distribution of the node position by leveraging a geometrically-derived unscented transformation of AoA estimates. The approximations are then integrated into a majority voting scheme, effectively eliminating outliers induced by multipath or other adverse effects. To derive the final enhanced position estimate, Bayesian inference is applied to fuse the selected information. Finally, the efficacy of the solution is validated conducting a comparative analysis against commonly used approaches in a real-world Bluetooth indoor localization system. The results obtained solely from high-level angular measurements underscore the practicality, robustness and high accuracy of the proposal.