MLED: A Layered Architecture for Reducing Undetected Error Probability in File Transfer

Abstract

Reliability in petabyte-scale file transfers is critical for data collected from scientific instruments. We introduce a Multi-Layer Error Detection (MLED) architecture that significantly reduces the Undetected Error Probability (UEP) in file transfer. MLED is parameterized by a number of layers $n$ , and a policy $P_{i}$ for each layer $1 \le i \le n$ that describes its operation. MLED generalizes existing error detection approaches used in file transfer. We show conditions under which MLED reduces UEP. Analytical results show that a petabyte-size file transfer in MLED with $n=2$ using CRC-32s improves UEP by $2.49E\,+\,28$ compared to a single-layer CRC-64, when the BER is 10−10.