Phenanthroline diamide ligands have been widely used in the separation of trivalent actinides and lanthanides, but little research has focused on extractants with asymmetrical substitutes. Two novel asymmetrical phenanthroline-based ligands N2,N2,N9-triethyl-N9-tolyl-1,10-phenanthroline-2,9-dicarboxamide (DE-ET-DAPhen) and N2-ethyl-N9,N9-dioctyl-N2-tolyl-1,10-phenanthroline-2,9-dicarboxamide (DO-ET-DAPhen) were first synthesized in this work, whose extraction ability and complexation mechanism to trivalent actinides [An(III)] and lanthanides [Ln(III)] were systematically investigated. The ligands dissolved in n-octanol exhibit good extraction ability and high selectivity toward Am(III) in acidic solutions. The complexation mechanism of the ligands with Ln(III) in solution and solid state was analyzed using slope analysis, 1H NMR spectrometric titration, ESI-MS, and calorimetric titration. It is revealed that the ligands complex with Am(III)/Eu(III) with 1:1 stoichiometry. The stability constant (log β) of the complexation reaction of Eu(III) with DE-ET-DAPhen determined by UV-vis spectrophotometric and calorimetric titration is higher than that of DO-ET-DAPhen, indicating the stronger complexation ability of DE-ET-DAPhen. Meanwhile, the calorimetric titration results show that the complexation process is exothermic with a decreased entropy. The structures of 1:1 complexes of Eu(III) and Nd(III) with DE-ET-DAPhen were analyzed through single-crystal X-ray diffraction. This work proves that ligands containing asymmetrical functional groups are promising for An(III)/Ln(III) separation, which shows great significance in efficient extractants designed for the spent nuclear fuel reprocessing process.