Solving real-world nonlinear semiconductor device problems modeled by the drift-diffusion equations coupled with the Poisson equation (also known as the Poisson-Nernst-Planck equations) necessitates an accurate and efficient numerical scheme which can avoid non-physical oscillations even for problems with extremely sharp doping profiles. In this paper, we propose a flexible and high-order hybridizable discontinuous Galerkin (HDG) scheme with harmonic averaging (HA) technique to tackle these challenges. The proposed HDG-HA scheme combines the robustness of finite volume Scharfetter-Gummel (FVSG) method with the high-order accuracy and hp-flexibility offered by the locally conservative HDG scheme. The coupled Poisson equation and two drift-diffusion equations are simultaneously solved by the Newton method. Indicators based on the gradient of net doping and solution variables are proposed to switch between cells with HA technique and high-order conventional HDG cells, utilizing the flexibility of HDG scheme. Numerical results suggest that the proposed scheme does not exhibit oscillations or convergence issues, even when applied to heavily doped and sharp PN-junctions. Devices with circular junctions and realistic doping profiles are simulated in two dimensions, qualifying this scheme for practical simulation of real-world semiconductor devices.