The tributyl phosphate (TBP)/FeCl3 system has been widely used for lithium extraction from high‑magnesium salt lakes. However, at high TBP concentrations, the Li/Mg selectivity is significantly lower. This study revealed that at high TBP concentrations, TBP alone extracted Mg2+ to form MgCl2·6TBP as the main product, thus affecting the extraction efficiency of Li+. On this basis, a neutral phosphate with long carbon chains, trioctyl phosphate (TOP), was used to replace TBP. The separation capacity and extraction mechanism of the TOP/FeCl3 system were studied, and density functional theory (DFT) calculations were employed to model the cationic complexes of the TOP and TBP systems. The results showed that in the presence of FeCl3, the extraction efficiency of Li+ (ELi) reached 91 %, and the Li/Mg separation factor (βLi/Mg) was 5688. In contrast, in the TBP/FeCl3 system with the same concentration, the ELi and βLi/Mg values were much lower at 74 % and 239, respectively. Furthermore, the TOP complex of Li+ had shorter Li − O bond lengths and a more negative binding energy, while that formed with Mg2+ had longer Mg − O bond lengths and the smallest HOMO–LUMO gap, indicating that TOP bonded more stably with Li+ and was less prone to Mg2+ extraction than TBP.