In this article, we explore the fundamental limits of the 3-dimensional (3D) localization of unmanned aerial vehicles (UAVs) in conjunction with the effects of 3D antenna radiation patterns. Although localization of UAVs has been studied to some extent in the literature, effects of antenna characteristics on 3D localization remains mostly unexplored. To study such effects, we consider a scenario where a fixed number of radio-frequency (RF) sensors equipped with single or multiple dipole antennas are placed at some known locations on the ground, and they derive the time-difference-of-arrival (TDOA) measurements from the time-of-arrival (TOA) data collected for the UAV that is also equipped with a dipole antenna. We then use these measurements to estimate the 3D location of the UAV, and to derive the Cramer-Rao lower bounds (CRLBs) on the localization error for various orientations of the dipole antennas at the transmitter and the receiver. Namely, we consider vertical-vertical (VV), horizontal-horizontal (HH), and vertical-horizontal (VH) radiation patterns in a purely line-of-sight (LoS) environment and a mixed LoS/Non-line-of-sight (NLoS) environment. We show that the localization accuracy changes in a non-monotonic pattern with respect to the UAV altitude and identify the respective critical altitudes for each of the VV, VH and HH orientations. Subsequently, we propose a multi-antenna signal acquisition technique that mitigates the accuracy degradation due to the antenna pattern mismatches, and we derive the localization CRLB for the multi-antenna scenario. Our numerical results characterize achievable localization accuracy for various antenna configurations, UAV heights, and propagation conditions for representative UAV scenarios.