Accurate trajectory prediction of surrounding agents is essential for autonomous vehicles, where the key challenge is to understand the complex interactions among agents. Previous works treat all interacted features between agents equally in modeling interaction, while neglecting their different importance to the interaction, thus inevitably limiting the interaction modeling ability. Besides, existing methods suffer from significant performance degradation when domain shifts, resulting in severely deviant prediction from reality. To address these issues, we propose a novel prediction framework, dubbed Mixing Feature Attention Network (MFAN). Specifically, the proposed mixing feature attention is a parallel design to adaptively determine the importance of different interacted features and simultaneously capture the global interaction feature to improve interaction modeling. Meanwhile, the spatial global interaction is modeled from a spatial edge-featured graph input to capture the enhanced spatial interaction. The temporal motion pattern is modeled from a temporal edge-featured graph input to enhance the domain adaption. Finally, we estimate the parameters of bivariant Gaussian distribution for trajectory prediction. Experimental results show that our method achieves superior performance in trajectory prediction while maintaining low computational complexity and performs accurate prediction even when domain shifts.