Uncovering the microscopic mechanism of low lattice thermal conductivity is essential for exploration and design of high-performance thermoelectrics. AgInSe2 exhibits high thermoelectric performance mainly due to its low thermal conductivity. Here, the origin of its intrinsic low lattice thermal conductivity is studied by temperature-dependent inelastic neutron scattering (INS), X-ray absorption fine structure (XAFS) spectra measurements, and first-principles calculations. A prominent “avoided crossing” feature and low-lying optical modes in the phonon dispersion of AgInSe2 are observed experimentally. These lattice dynamical features cause a local reduction of the phonon group velocity and strongly scatter heat-carrying acoustic phonons, contributing to its intrinsic low lattice thermal conductivity. In addition, both temperature-dependent phonon dispersions and phonon density-of-states measurements reveal strong anharmonicity or phonon-phonon interactions in AgInSe2. XAFS and phonon eigenvector analysis demonstrate the dominant role of Ag vibrations, which is closely associated with the “avoided crossing”, low-lying optical modes and large structural distortion, and thus dominates the reduction of lattice thermal conductivity of AgInSe2.