The innovative mechanism of steady-state microbunching (SSMB) promises a potent light source, featuring high repetition rate and coherent radiation. The laser modulator, comprising an undulator and an optical enhancement cavity, is pivotal in SSMB. A high-finesse prototype optical enhancement cavity for SSMB with an average power of 55 kW is described in this paper. Preliminary design of the laser modulator, experimental setup, and methods to address frequency degeneracy and power coupling issues are discussed. D-shaped mirrors are utilized to successfully suppress the modal instability. This study is the first to illustrate the finesse reduction caused by high-order mode damping during experiments. The experimental and simulation results match closely. A cavity power coupling model is established, and the experimental results verify the correctness of the coupling model. A method for estimating the absorption coefficient through thermal-induced evolution of cavity mode has been implemented. Experimental results demonstrate a high-average-power enhancement cavity with a finesse of 16 518 ± 103 and an estimated average absorption coefficient of 12 ppm for the cavity mirrors. The findings contribute to the advancement of SSMB by providing insights into the design and operation of high-power optical enhancement cavities.