By extending the electroweak gauge group to $SU(3)_L\times U(1)_Y$, the 331 model incorporates dilepton gauge bosons $Y$ which do not respect individual lepton family number. We point out that, in addition to family diagonal couplings such as $Y$--$e$--$e$ that change lepton family number by two units, dileptons may also have family non-diagonal couplings such as $Y$--$\mu$--$e$. The latter coupling violates lepton family number by a single unit and manifests itself via lepton flavor changing decays such as $\mu\to3e$ and $\mu\to e\gamma$. The family non-diagonal interaction can be CP violating and typically generates extremely large leptonic electric dipole moments. We demonstrate a natural mechanism for eliminating both single unit lepton flavor violation and large leptonic CP violation. Although we focus on the 331 model, our results are applicable to other dilepton models as well, including $SU(15)$ grand unification.