In recent years at Tencent, we have observed that the use of multiple NoSQL databases for storing business data with diverse models has led to increased programming and deployment costs, as well as inefficient maintenance and underutilized resources. In this paper, we report X-Stor, a cloud-native NoSQL database system that supports multiple data models by extending different storage engines and efficiently managing them through a unified control plane. This design significantly reduces expenses and enables rapid expansion of new models, while seamlessly supporting their complete functionality through storage engine extensions. By consolidating multi-tenant services and data models on the same physical machines, X-Stor significantly enhances the utilization of cluster resources. Additionally, X-Stor introduces a standardized metric called Request Unit (RU) to measure tenant resource consumption for consumption-based pricing purposes. Leveraging this metric, we design RU-based resource management strategies and achieve efficient multi-tenant resource isolation and system load balancing. Currently, X-Stor manages a storage capacity of over 12PB for online operational data, including more than 100,000 tables with multiple data models. It handles 700 billion requests per day with a peak of 30 million requests per second. We evaluate the performance of X-Stor on popular benchmarks and production workloads. The results show that X-Stor performs well under diverse data models.
Read full abstract