Lepton number as a fourth color is an intriguing theoretical idea which is combined with a possible left-right symmetry within the famous Pati-Salam (PS) model. In the conventional PS model, a spontaneous breaking of the PS gauge group down to the SM one can only take place at very high scales (above the PeV scale) due to the stringent bounds from $K_L\to\mu e$ and $K\to\pi\mu e$ induced by the resulting vector leptoquarks. In this paper, we show that these constraints can be avoided once additional vector-like fermions are introduced and thus a breaking at the TeV scale is possible. We consider the flavor phenomenology of this model in the context of the intriguing hints for new physics in semileptonic $B$ decays. The necessary violation of lepton flavor universality is induced by mixing SM and vector-like fermions. Concerning $R(D)$ and $R(D^*)$ we find that sizable effects are possible while respecting the bounds from other flavor observables but predicting a large enhancement of $B_s\to\tau^+\tau^-$. Furthermore, also in $b\to s\ell^+\ell^-$ transitions the observed deviations from the SM predictions (including $R(K)$ and $R(K^*)$) can be explained with natural values for the free parameters of the model without any fine-tuning, predicting sizable decay rates for $b\to s\tau\mu$. Finally, the anomaly in anomalous magnetic moment of the muon can be accounted for by a loop-contribution involving the vector leptoquark and vector-like leptons.