High Performance Distributed In-Memory Architectures for Trade Surveillance System

Abstract

With rapid growth of world economy, the user activity in the capital markets is increased. This results into large number of transactional activities in trading systems. Hence, the trade surveillance system with low latency and high throughput is needed to monitor such a large amount of data in order to improve user experience by reducing discrepancies and frauds. In-memory technology reduces this latency by processing as well as caching data in main memory thereby removing the overhead of disk access. Currently, open-source frameworks such as Apache Ignite, Apache Flink and Kafka Streams provides in-memory streaming and caching functionalities along with scalability and fault-tolerant features. The paper talks about Trade Surveillance System (TSS), which includes Complex Event Processing (CEP). Here we discuss design, implementation and tuning of three different high-performance architectures for trade surveillance system using Ignite, Flink and Kafka Streams as in-memory streaming technologies. Paper also compares system throughput, support for fault tolerance and effect of caching on streaming throughput for all three architectures. Based on experiments, it is seen that Ignite outperforms Flink and Kafka Streams in CEP based streaming. Flink is more reliable considering fault-tolerance and event-time processing at streaming layer compared to Ignite. Though Kafka Streams also provides fault-tolerance and event-time processing out of the box, it shows high latency due to disk based processing.