Private key (e.g., RSA key) protection is a significant issue for cloud but existing keyless or keyguard solutions suffer from performance, elasticity or applicability limitations. Recently, represented by Intel KPT, a novel keyguard architecture emerges to combine trusted platform module and crypto accelerator for achieving both security and performance. However, the straight use of KPT for private key protection may not be a good fit in cloud as it incurs challenges on protection capacity, key provisioning latency and transparency. Based on KPT-like hardware, we propose QKPT, a comprehensive key management system to bring your own private keys (BYOPK) into multi-tenant clouds. QKPT introduces a carefully-designed key wrapping layer to overcome these challenges. A small symmetric wrapping key (SWK) is generated for each tenant as the master key to resolve the former two challenges, while a special private key wrapping scheme is adopted to resolve the transparency limitation. Additionally, QKPT incorporates certificate trust to enhance the security of the SWK lifecycle and provides a hardened key server solution without expensive HSM. The evaluation shows that QKPT has a low runtime overhead ( <inline-formula><tex-math notation="LaTeX">$\leq$</tex-math></inline-formula> 1.2% for SSL/TLS handshakes) and still greatly outperforms the software baseline (3.5x-17x) owing to the crypto offloading.