We report on a pure $^{181}$Ta-nuclear quadrupole resonance (NQR) measurement of PbTaSe$_2$ at zero magnetic field, which has the advantage of directly probing the intrinsic superconducting phase and electronic states of the TaSe$_2$ layer. We observed the $^{181}$Ta-NQR spectrum of the intrinsic structure with space group $P6$-$m2$, which agrees well with density functional theory (DFT) calculations. The nuclear spin relaxation rate ($1/T_1$) shows an exponential decrease well below $T_{\rm_c}$, indicating that the superconducting state is fully gapped in the framework of Bardeen-Cooper-Schrieffer (BCS) theory. The gap size obtained by $^{181}$Ta-NQR was smaller than the value in previous reports, which may imply that the Fermi surfaces composed of Ta-5$d$ orbitals, where the average pairing interactions are expected to be weaker than in BCS model, are primarily probed. The temperature dependence of $1/T_1$ below $T_{\rm_c}$ can be reproduced well by the superposition of quadrupole and magnetic relaxation mechanisms, together with the distribution of superconducting gap size inherent to multiple Fermi surfaces theoretically proposed in PbTaSe$_2$.