The pressure dependence of the upper critical field, $H_\textrm{c2,c}$, of single crystalline FeSe was studied using measurements of the inter-plane resistivity, $\rho_{\textrm{c}}$ in magnetic fields parallel to tetragonal $c$-axis. $H_\textrm{c2,c}(T)$ curves obtained under hydrostatic pressures up to $1.56$ GPa, the range over which the superconducting transition temperature, $T_\textrm{c}$, of FeSe exhibits a non-monotonic dependence with local maximum at $p_1\approx$ 0.8 GPa and local minimum at $p_2\approx$ 1.2 GPa. The slope of the upper critical field at $T_\textrm{c}$, $\left(\textrm{d}H_\text{c2,c}/\textrm{d}T\right)_{T_\textrm{c}}$, also exhibits a non-monotonic pressure dependence with distinct changes at $p_1$ and $p_2$. For $p<p_1$ the slope can be described within multi-band orbital model. For both $p_1<p <p_2$ and $p>p_2$ the slope is in good quantitative agreement with a single band, orbital Helfand-Werthamer theory with Fermi velocities determined from Shubnikov-de Haas measurements. This finding indicates that Fermi surface changes are responsible for the local minimum of $T_\textrm{c}(p)$ at $p_2\approx$ 1.2 GPa.