Incentive-based integrated demand response has great potential in addressing supply-demand imbalance in integrated energy system. This study first proposes an incentive-based integrated demand response model based on multi-agent reinforcement learning, to deal with the multi energy coupling on the demand-side and enhance the applicability of the incentive-based demand response program in urban integrated energy system. Then, a self-adaptive method is proposed based on consumer energy consumption behavior, which utilizes nonlinear incentive strategy to encourage consumers to provide more demand response resources. Finally, the experimental results show that considering demand-side coupling can enhance the overall benefits of incentive-based integrated demand response, achieving a win-win situation between integrated energy service provider and consumers. After implementing incentive strategy that considers demand-side coupling, profit of integrated energy service provider increased by 16.27% and the discomfort costs borne by integrated energy consumers reduced by 11.5%. This study offers a promising approach to managing the complex and variable energy consumption behaviors in urban integrated energy system, contributing to more reliable and efficient operation of urban integrated energy system.