In recent years, materials with metal-to-insulator phase transitions have attracted great attention due to their property variations before and after the transition. As nearly all good thermoelectric materials are small band-gap semiconductors, a metal-insulator transition makes a profound difference in thermoelectric performance. Revealing the microscopic reasons for these variations is critical to understanding the structure-property relation in these materials. MgAgSb is an important thermoelectric material around room temperature, due to its high zT value and intrinsic low lattice thermal conductivity in the semiconducting α-phase. However, the thermoelectric properties of mid-temperature β-phase MgAgSb significantly deteriorate because the β-phase has no band gap. In this work, we reveal that the Ag atomic rearrangement between the two phases is the major reason responsible for the metal-insulator transition and deterioration of thermoelectric properties. In MgAgSb, there are strong interactions between Mg and Sb, as well as Ag and Sb, leading to bands of mixed character around the Fermi Level. In α-MgAgSb, the Mg–Sb bands in the conduction band minimum separate well with the bands around the valence band maximum, and form a finite band gap. Due to the Ag rearrangement and the formation of a quasi-two-dimensional structure of β-MgAgSb, the interactions among Mg–Ag–Sb atoms form multiple bands crossing the Fermi level, and realize the “two-dimensional metallization”. In addition, the structural variations in the β-phase results in higher phonon velocities and removal of additional low-frequency optical phonons as only shown in the α-phase, both of which leading to the relatively high κL of the β-phase. Our work provides new perspectives for the performance research and application of materials with metal-insulator phase transitions.