To overcome the drawbacks such as emulsification and esterification reaction of the extractant and phase modifiers of the commonly used naphthenic acid (NA) system for yttrium (Y) separation, some novel extractants have to be developed. By combining the functional groups of commercial organophosphorus extractants, carboxylic acids and alcohol phase modifiers, a novel extractant 2-(bis((2-ethylhexyl)oxy)phosphoryl)-2-hydroxyacetic acid (HPOAc, HL) was designed and synthesized. The pKa value of HPOAc is a little larger than that of 2-ethylhexyl phosphoric acid mono-2-ethylhexylester (P507) and smaller than that of common carboxylic acid extractants such as NA and sec-octylphenoxy acetic acid (CA−12). Y(III) is the most difficult to be extracted among trivalent rare earth elements (REEs(III)) by HPOAc. The distribution ratios exhibit a distinct “tetrad effect”. The extraction mechanism follows a cation exchange mechanism with the extraction complex being deduced as REL3. Compared with NA and CA−12, HPOAc has higher extraction ability for REEs(III), and better performance in the separation of Ln(III) and Y(III). The extraction capacity for Y(III) is about 1.75 times higher than those of NA and CA−12. The loaded Y(III) and Sm(III) can be completely stripped by 0.75 mol/L HCl solution. Moreover, even if the organic phase is saturated or there are several impurity metal ions (Mg(II), Al(III), Ca(II), Mn(II) and Zn(II)) in the aqueous solution, two phases can still be separated very quickly and the fluidity of the loaded organic phase is good without the phase modifiers. Encouragingly, the HPOAc system also exhibits good separation performance towards REEs(III) and the impurity metal ions. Finally, a cascade extraction process for separating Ln(III) and Y(III) was developed using HPOAc as the extractant. The Y product with the purity of 99.31 wt% was obtained by 20 stages of extraction and 4 stages of scrubbing.