Controlling the leakage flow of the labyrinth seal cavity is very important for the efficiency and aerodynamic matching design of aero-engine high-pressure compressor. However, due to the complexity of the geometry and internal flow of the labyrinth seal cavity, the design optimization of the labyrinth seal cavity is rarely studied. In this paper, a multi-objective design optimization method was used to optimize the stationary endwall of the shrouded stator labyrinth cavity, and the optimal endwall shape was obtained. Based on the optimal geometry, the 3D printing and aerodynamic testing of experimental parts were carried out, and the aerodynamic performance and flow field details of the optimal geometry were analyzed by unsteady numerical simulation. The experimental results show that for the optimal geometry, it has a little effect on the total pressure ratio, but the efficiency at the design point is increased by 1.4 %. The numerical results show that the optimal cavity shape reduces low-energy fluid, thus reduces the total pressure loss of S3. At the near stall point, the optimal S3 still has a better performance, which means that the optimal shape is not limited to the design point. From the result, an engineering design criterion can be concluded. It is suggested that the labyrinth seal cavity should be designed to match the velocity components of the leakage flow with the mainstream of the stator properly to make the leakage flow go evenly in the blade passage instead of gathering to the suction side.