AIDTN: Towards a Real-Time AI Optimized DTN System With NVMeoF

Abstract

Large-scale data transport for data-intensive sciences is a complex multidimensional challenge. The challenge includes optimizing the end-to-end Big Data movement performance in real-time, supporting direct remote data access using NVMe over Fabrics (NVMeoF) and deploying to existing research platforms. AIDTN is the first effort to provide a unique AI system designed to incorporate NVMe over Fabrics (NVMeoF) and optimize coordination among multiple components supporting large-scale, multi-domain Wide Area Network (WAN) data-intensive science. AIDTN's research objective is to integrate next-generation storage architecture using NVMeoF, specialized network design using high-performance network appliances, Data Transfer Nodes (DTNs), catalysts in driving data transport, and a unique AI system explicitly designed for high-performance data movement challenges. AIDTN is the first system that uses network and system features to predict the end-to-end performance of high-performance data movement and further extends the model with NVMe-specific features for NVMeoF remote data access. As a result, AIDTN improves data movement performance by up to 284% while minimizing packet loss compared to other heuristics approaches. It also has a prediction error rate as low as 0.16 compared to AI models with the only network (error rate = 0.29) or network and system features (error rate = 0.19).