Zatara, the Plug-in-able Eventually Consistent Distributed Database

Abstract

With the proliferation of the computer Cloud, new software delivery methods were created. In order to build software to fit into one of these models, a scalable, easy to deploy storage tier is required. Distributed, non-SQL databases use multiple techniques to distribute information, guarantee data consistency and grow, but unfortunately most developments were designed with a single class of applications in mind, which means that they bring many constraints for developers. Existent solutions range from simple key-value databases to more complex approaches usually developed for data indexing. Our approach is a multi-purpose, distributed database engine that features an abstract query interface and plug-in-able internal data structures. The database, called ZATARA, is tested on an Amazon EC2 infrastructure with 196 nodes. In this environment it is able to deliver more than 20 million transactions per second, scaling almost linearly with the number of nodes. From the performance point of view, these results demonstrate our initial assumptions that it is not necessary to expose a particular data structure in order for the database to scale. Although Zatara cannot replace SQL databases in all deployments, it provides sufficient flexibility to make it a viable choice for most applications that have to scale indefinitely. Also, it can be used as a caching system in order to reduce load on the SQL tier, in these scenario being trivial to add it to an existing application.