• A mathematical model for the vapor-bypassed evaporator is developed. • Corresponding experiments were conducted to validate the model accuracy. • Entropy generation is applied to evaluate the evaporator overall performance. • Phase separation position and channel number of the evaporator are optimized. This paper presents a simulation study on the vapor-bypassed evaporator for heat pump applications. The vapor-bypassed evaporator, in which a phase separator is set in the middle of the refrigerant circuit to bypass vapor refrigerant, can reduce the refrigerant pressure drop for enhancing the evaporator overall performance. A mathematical model for the vapor-bypassed evaporator is developed and the experiments were conducted to validate its accuracy. Based on the entropy generation minimization theory, the optimum phase separation positions and refrigerant channel numbers of the evaporator for various operation conditions are selected out. According to the simulation results, as the required heat transfer capacity ranges 4~10 kW, the vapor-bypassed (VB) mode obtains 25.6~84.6 kPa lower refrigerant pressure drop and 3.1~25.3% lower entropy generation than the conventional operation (CO) mode. Moreover, when the ambient temperature ranges 5~15°C, the entropy generation of VB mode is decrease by 22.3~11.5% than CO mode. In general, applying the vapor-bypassed technique is a promising way for improving the evaporator performance especially under high heat transfer capacity and low ambient temperature conditions.