We investigate constraints on the new $B-L$ gauge boson ($Z_{BL}$) mass and coupling ($g_{BL}$) in a $U(1)_{B-L}$ extension of the standard model (SM) with an SM singlet Dirac fermion ($\zeta$) as dark matter (DM). The DM particle $\zeta$ has an arbitrary $B-L$ charge $Q$ chosen to guarantee its stability. We focus on the small $Z_{BL}$ mass and small $g_{BL}$ regions of the model, and find new constraints for the cases where the DM relic abundance arises from thermal freeze-out as well as freeze-in mechanisms. In the thermal freeze-out case, the DM coupling is given by $g_{\zeta}\equiv g_{BL}Q\simeq0.016\sqrt{m_\zeta[{\rm GeV}]}$ to reproduce the observed DM relic density and $g_{BL}\geq 2.7 \times 10^{-8} \sqrt{m_\zeta[{\rm GeV}]}$ for the DM particle to be in thermal equilibrium prior to freeze-out. Combined with the direct and indirect DM detection constraints, we find that the allowed mass regions are limited to be $m_\zeta \gtrsim 200$ GeV and $M_{Z_{BL}} \gtrsim 10$ GeV. We then discuss the lower $g_{BL}$ values where the freeze-in scenario operates and find the following relic density constraints on parameters depending on the $g_{BL}$ range and dark matter mass: Case (A): for $g_{BL}\geq 2.7\times10^{-8}\sqrt{m_\zeta[{\rm GeV}]}$, one has $g^2_\zeta\,g^2_{BL}+\frac{0.82}{1.2}\,g^4_\zeta\simeq 8.2\times10^{-24}$ and Case (B): for $g_{BL} < 2.7 \times 10^{-8} \sqrt{m_\zeta[{\rm GeV}]}$, there are two separate constraints depending on $m_\zeta$. Case (B1): for $m_\zeta\lesssim 2.5{\rm TeV}$, we find $g_\zeta^2\,g_{BL}^2\simeq 8.2\times10^{-24}\,\left( \frac{m_\zeta}{2.5\,{\rm TeV}} \right)$ and case (B2): for $m_\zeta \gtrsim 2.5$ TeV, we have $g_\zeta^2 \, g_{BL}^2 \simeq 8.2 \times 10^{-24}$. For this case, we display the various parameter regions of the model that can be probed by a variety of ``Lifetime Frontier" experiments such as FASER, FASER2, Belle II, SHiP and LDMX.