In order to adequately leverage the advantages of coaxial compound helicopters (CCH) to enhance overall performance and efficiency through variable rotor speed, and to improve the fuel economy of the composite propulsion system, a real-time optimization control method of variable rotor speed based on helicopter/turboshaft engine on-board composite system is proposed. Firstly, a steady-state model of the CCH required power and an on-board inverse model of the turboshaft engine based on an improved deep neural network propulsion system model (AdamDNN-PSM) are established. On this basis, an on-board adaptive model based on adaptive square root Kalman filter (ASRUKF) estimator is employed to correct the precision of the on-board inverse model and to track the real engine state. Finally, by adopting the feasible sequence quadratic programming (FSQP) method, an optimal speed rotor (OSR) is obtained through real-time optimization control, which reduces fuel consumption during cruising while considering both the geometric and aerodynamic constraints of the rotor. The simulation results demonstrate that the accuracy and real-time performance of the on-board inverse model have been significantly improved. The real-time optimization control method for OSR effectively reduces fuel consumption during cruising and enhances the CCH performance of the entire flight envelope.