In our recent attempt to explain flavor hierarchies [1], a gauged SU(2) flavor symmetry acting on left-handed fermions provides a ground to introduce three independent rank-one contributions to the Yukawa matrices: a renormalizable one for the third family, a mass-suppressed one for the second family, and an additional loop-suppressed factor for the first family. Here, we demonstrate how minimal quark-lepton unification à la Pati-Salam, relating down-quarks to charged leptons, can significantly improve this mechanism. We construct and thoroughly analyze a renormalizable model, performing a comprehensive one-loop matching calculation that reveals how all flavor hierarchies emerge from a single ratio of two scales. The first signatures may appear in the upcoming charged lepton flavor violation experiments.