Manipulating thermal properties of materials can be interpreted as the control of how vibrations of atoms (known as phonons) scatter in a crystal lattice. Compared to a perfect crystal, crystalline solids with defects are expected to have shorter phonon mean free paths caused by point defect scattering, leading to lower lattice thermal conductivities than those without defects. While this is true in many cases, alloying can increase the phonon mean free path in the Cd-doped AgSnSbSe3 system to increase the lattice thermal conductivity from 0.65 to 1.05Wm-1K-1 by replacing 18% of the Sb sites with Cd. It is found that the presence of lone pair electrons leads to the off-centering of cations from the centrosymmetric position of a cubic lattice. X-ray pair distribution function analysis reveals that this structural distortion is relieved when the electronic configuration of the dopant element cannot produce lone pair electrons. Furthermore, a decrease in the Grüneisen parameter with doping is experimentally confirmed, establishing a relationship between the stereochemical activity of lone pair electrons and the lattice anharmonicity. The observed "harmonic" behavior with doping suggests that lone pair electrons must be preserved to effectively suppress phonon transport in these systems.