Striving to develop renewable energy generation (REG) has become a global trend. Due to high carbon emissions, fossil-fueled generation units (FFGUs) in an electrical energy market such as the day-ahead and real-time electricity markets are facing unprecedented constraints. From the perspective of FFGUs, an effective way is to change their market roles from energy suppliers with high environmental costs to secure and reliable frequency regulation service providers. To help FFGUs successfully complete the transition, a bi-level optimization model for the joint energy and regulation service market is proposed in this paper. Different from existing joint markets, both the environmental cost of FFGUs and the frequency regulation service (FRS) cost of REG units are considered in this research for market operators to determine the scheduling priorities of submitted bids from generation units. In the frequency regulation service market model, a dynamic approach to determining the FRS demand is applied, which not only incentivizes REG units to initiatively manage their outputs, but also derives the price signals through the equilibrium of the FRS demand and supply to ensure the revenue of FFGUs in the FRS market. Meanwhile, the decision-making model of the FFGUs under the proposed market mechanism is also presented. Finally, extensive numerical experiments demonstrate the feasibility and efficiency of the mechanism. Simulation results show that the proposed mechanism can stimulate a smooth transition of FFGUs by adapting to actual situations and policy changes.