In this paper, we develop a prisoner's dilemma game model with three types of strategies and six types of emotions. We propose a direct emotional interaction mechanism considering memory effect in which a player's favorability serves as an index of emotions and directly impacts payoffs and emotions. Our model enables that individuals gradually increase the tendency to loneliness or defection in the face of unfriendly people, and increase the tendency to cooperation in the face of friendly people. By simulations, we further show that both direct emotional interaction and memory have the effect of enhancing the competitive advantage of cooperators and individuals with friendly emotions. Besides, the proportion of friendly groups increases if the degree of direct emotional interaction increases. Also, the impact of memory on cooperation has a non-monotonic characteristic. As a result, direct emotional interaction with memory effect leads to the improvement of social average favorability, increases the number of friendly populations, improves the overall level of social cooperation, and enables friendly people to gain advantages in evolution. Therefore, our work contributes to understanding the role of emotional factors in the evolution of social cooperation.