ABSTRACT This paper proposes a game theory-based on-ramp merging controller for connected automated vehicles (CAVs) in mixed traffic flow. First, a two-layer decision-making framework based on the Stackelberg game is designed to consider the fuel consumption and safety payoffs of mixed traffic flow under different driving behaviors. The upper layer of the framework determines the optimal merging decision (i.e. merging time and location) for on-ramp vehicles (RVs) based on the Stackelberg game. The lower layer optimizes the merging trajectory of CAVs to reduce energy consumption and safety risks during the ramp-merging process. Then, a driving behavior estimation algorithm is developed to describe the differences in mainline vehicles (MLVs) response to the merging behavior of RVs. Finally, the simulation experiments are adopted to verify the effectiveness and stability of the proposed framework. The results indicated that, the proposed framework promotes environmental protection, operational efficiency, and traffic flow stability in different traffic scenarios.