Optimization model based on generalized equivalent uniform dose (gEUD) linear sub-score or quadratic sub-score, which only penalizes doses higher or lower than the prescribed dose gEUD0, has the shortcomings of semi-deviation, vanishing gradient and non-increasing in the feasible solution space. When gradient-based optimization algorithms are used to solve the radiotherapy inverse optimization problem, these algorithms may get trapped in a local minimum. To address these drawbacks, this study proposes a new gEUD-based optimization model based on regularization theory. In the new optimization model, a dosage whether lower or higher than the prescribed dose is assigned different penalties. To test its efficiency, it was tested on a phantom TG119, and two types of clinic cases(one prostate cancer case and one head and neck cancer case). The improved optimization model was compared with unimproved gEUD-based optimization model. Additionally, the improved gEUD-based optimization model was compared with another improved gEUD-based linear optimization model proposed by us. The gradient-based optimization algorithm(L-BFGS) was applied to solve these large-scale optimization problems. Optimization based on our improved optimization model is capable of improving the organs at risk (OARS) sparing while maintaining the same planning target volume (PTV) coverage. In practice, although the DV-based optimization should be able to gain a similar plan, parameters adjustment of the optimization model is time-consuming. The new gEUD-based hybrid physical–biological optimization model has the potential to expand the solution space and improve the quality of radiotherapy plan.