We discuss in detail the shear viscosity coefficient $\ensuremath{\eta}$ and the viscosity to entropy density ratio $\ensuremath{\eta}/s$ of a hadronic gas comprised of pions and nucleons. In particular, we study the effects of baryon chemical potential on $\ensuremath{\eta}$ and $\ensuremath{\eta}/s$. We solve the relativistic quantum Boltzmann equations with binary collisions ($\ensuremath{\pi}\ensuremath{\pi}$, $\ensuremath{\pi}N$, and $NN$) for a state slightly deviated from thermal equilibrium at temperature $T$ and baryon chemical potential $\ensuremath{\mu}$. The use of phenomenological amplitudes in the collision terms, which are constructed to reproduce experimental data, greatly helps to extend the validity region in the $T\mathrm{\text{\ensuremath{-}}}\ensuremath{\mu}$ plane. The total viscosity coefficient $\ensuremath{\eta}(T,\ensuremath{\mu})={\ensuremath{\eta}}^{\ensuremath{\pi}}+{\ensuremath{\eta}}^{N}$ increases as a function of $T$ and $\ensuremath{\mu}$, indirectly reflecting energy dependences of binary cross sections. The increase in $\ensuremath{\mu}$ direction is due to enhancement of the nucleon contribution ${\ensuremath{\eta}}^{N}$ while the pion contribution ${\ensuremath{\eta}}^{\ensuremath{\pi}}$ diminishes with increasing $\ensuremath{\mu}$. On the other hand, due to rapid growth of entropy density, the ratio $\ensuremath{\eta}/s$ becomes a decreasing function of $T$ and $\ensuremath{\mu}$ in a wide region of the $T\mathrm{\text{\ensuremath{-}}}\ensuremath{\mu}$ plane. In the kinematical region we investigated $T<180\text{ }\text{ }\mathrm{MeV}$, $\ensuremath{\mu}<1\text{ }\text{ }\mathrm{GeV}$, the smallest value of $\ensuremath{\eta}/s$ is about 0.3. Thus, it never violates the conjectured lower bound $\ensuremath{\eta}/s=1/4\ensuremath{\pi}\ensuremath{\sim}0.1$. The smallness of $\ensuremath{\eta}/s$ in the hadronic phase and its continuity at $T\ensuremath{\simeq}{T}_{c}$ (at least for crossover at small $\ensuremath{\mu}$) implies that the ratio will be small enough in the deconfined phase $T\ensuremath{\gtrsim}{T}_{c}$. There is a nontrivial structure at low temperature and at around normal nuclear density. We examine its possible interpretation as the liquid-gas phase transition.