This paper presents a modified fifth-order WENO-HLLC Riemann solver for the transonic compressible viscous flows around the helicopter rotor in hover. The HLLC approximate Riemann solver is proposed to discrete the convection term involving grid velocity of the Navier-Stokes equations. In order to solve the interface flow accurately, a modified fifth-order WENO scheme is presented by designing the smoothness indicators based on L 1 norm measurement. The improved WENO scheme can provide the optimal approximation order even at critical points. Numerical accuracy and robustness are validated by several benchmark inviscid flow problems. Then the numerical properties of the WENO-HLLC solver in conjunction with the implicit LU-SGS time integration method with high efficiency are further validated by simulating transonic viscous flows over RAE2822 airfoil and ONERA-M6 wing. The results show that the accuracy of calculating shock, discontinuity, and the vortex is significantly improved. Finally, the method is developed to compute the transonic vortex flow around the helicopter rotor with a domain discretized by overset grids. The results indicate that the proposed method is very robust and effective in acquiring high resolution for vortex wake.