The long-range periodic atomic arrangement or the lack thereof in solids typically dictates the magnitude and temperature dependence of their lattice thermal conductivity (κlat). Compared to crystalline materials, glasses exhibit a much-suppressed κlat across all temperatures as the phonon mean free path reaches parity with the interatomic distances therein. While the occurrence of such glass-like thermal transport in crystalline solids captivates the scientific community with its fundamental inquiry, it also holds the potential for profoundly impacting the field of thermoelectric energy conversion. Therefore, efficient manipulation of thermal transport and comprehension of the microscopic mechanisms dictating phonon scattering in crystalline solids are paramount. As quantized lattice vibrations (i.e., phonons) drive κlat, atomistic insights into the chemical bonding characteristics are crucial to have informed knowledge about their origins. Recently, it has been observed that within the highly symmetric 'averaged' crystal structures, often there are hidden locally asymmetric atomic motifs (within a few Å), which exert far-reaching influence on phonon transport. Phenomena such as local atomic off-centering, atomic rattling or tunneling, liquid-like atomic motion, site splitting, local ordering, etc., which arise within a few Å scales, are generally found to drastically disrupt the passage of heat carrying phonons. Despite their profound implication(s) for phonon dynamics, they are often overlooked by traditional crystallographic techniques. In this review, we provide a brief overview of the fundamental aspects of heat transport and explore the status quo of innately low thermally conductive crystalline solids, wherein the phonon dynamics is majorly governed by local structural phenomena. We also discuss advanced techniques capable of characterizing the crystal structure at the sub-atomic level. Subsequently, we delve into the emergent new ideas with examples linked to local crystal structure and lattice dynamics. While discussing the implications of the local structure for thermal conductivity, we provide the state-of-the-art examples of high-performance thermoelectric materials. Finally, we offer our viewpoint on the experimental and theoretical challenges, potential new paths, and the integration of novel strategies with material synthesis to achieve low κlat and realize high thermoelectric performance in crystalline solids via local structure designing.